Seg faults when querying performance counters

Question

[reported by blelbach] [Trac time Thu Oct 13 02:35:29 2011] Querying counters breaks for all examples, doesn't seem to matter what counter I query.

Hartmut, when did you last test performance counters?

[02:30:21]:wash:/home/wash/hpx/gcc-4.6.1-release$ bin/hello_world --print-counter '/counters/time(locality#1/total)/idle-rate' --print-counter '/counters/time(locality#1/os-thread#0)/idle-rate'
hello world from OS-thread 0 on locality 1
Received Segmentation fault, 11 frames:
0x7fd7badde365  : ??? + 0xbadde365 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7badd8fe9  : hpx::detail::backtrace() + 0x9 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7bada15d6  : hpx::termination_handler(int) + 0x16 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7b99ad020  : ??? + 0xb99ad020 in /lib/x86_64-linux-gnu/libpthread.so.0
0x7fd7bb01fdcc  : hpx::performance_counters::stubs::performance_counter::get_value(hpx::naming::id_type const&, hpx::error_code&) + 0xc in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7bb06eba8  : hpx::util::query_counters::evaluate() + 0x1a8 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7baf58594  : hpx::components::server::runtime_support::call_shutdown_functions(bool) + 0x34 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7baf8102f  : ??? + 0xbaf8102f in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7baf7a07b  : ??? + 0xbaf7a07b in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7bae62da6  : ??? + 0xbae62da6 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7bae63169  : ??? + 0xbae63169 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
Aborted

[02:31:16]:wash:/home/wash/hpx/gcc-4.6.1-release$ bin/hello_world --print-counter '/counters/time(locality#1/total)/idle-rate' --print-counter-interval 10

[stack_trace]: 9 frames:
0x7fd7f7c3e365  : ??? + 0xf7c3e365 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7c38fe9  : hpx::detail::backtrace() + 0x9 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7c4a141  : void hpx::detail::throw_exception<:exception>(hpx::exception const&, std::string const&, std::string const&, int) + 0x41 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7ecf2ef  : hpx::util::query_counters::start() + 0x1cf in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7db8514  : hpx::components::server::runtime_support::call_startup_functions() + 0x34 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7c58d59  : ??? + 0xf7c58d59 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7c0bbc8  : ??? + 0xf7c0bbc8 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7cc2da6  : ??? + 0xf7cc2da6 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7fd7f7cc3169  : ??? + 0xf7cc3169 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
[what]: unknown performance counter: '/counters/time(locality#1/total)/idle-rate': HPX(bad_parameter)
[locality]: 1
[function]: query_counters
[file]: /home/wash/hpx/src/util/query_counters.cpp
[line]: 32
[os-thread]: 0
[thread_id]: 0000000002112018
[thread_name]: hpx::components::server::runtime_support::call_startup_functions_action
[version]: V0.7.0-trunk (AGAS: V2.1), SVN: 5953:5954M
[boost]: V1.47.0
[build-type]: release
[date]: Oct 13 2011 01:27:46
[platform]: linux
[compiler]: GNU C++ version 4.6.1
[stdlib]: GNU libstdc++ version 20110714

Aborted
</:exception>

[02:33:11]:wash:/home/wash/hpx/gcc-4.6.1-release$ bin/hello_world --print-counter '/counters/parcelqueue(locality#1/total)/length/instantaneous'
hello world from OS-thread 0 on locality 1
Received Segmentation fault, 11 frames:
0x7f13b491a365  : ??? + 0xb491a365 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b4914fe9  : hpx::detail::backtrace() + 0x9 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b48dd5d6  : hpx::termination_handler(int) + 0x16 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b34e9020  : ??? + 0xb34e9020 in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f13b4b5bdcc  : hpx::performance_counters::stubs::performance_counter::get_value(hpx::naming::id_type const&, hpx::error_code&) + 0xc in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b4baaba8  : hpx::util::query_counters::evaluate() + 0x1a8 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b4a94594  : hpx::components::server::runtime_support::call_shutdown_functions(bool) + 0x34 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b4abd02f  : ??? + 0xb4abd02f in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b4ab607b  : ??? + 0xb4ab607b in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b499eda6  : ??? + 0xb499eda6 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f13b499f169  : ??? + 0xb499f169 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
Aborted

[02:33:39]:wash:/home/wash/hpx/gcc-4.6.1-release$ bin/hello_world --print-counter '/counters/parcelqueue(locality#1/total)/length/instantaneous' --print-counter-interval 1

[stack_trace]: 9 frames:
0x7f4b713ac365  : ??? + 0x713ac365 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b713a6fe9  : hpx::detail::backtrace() + 0x9 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b713b8141  : void hpx::detail::throw_exception<:exception>(hpx::exception const&, std::string const&, std::string const&, int) + 0x41 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b7163d2ef  : hpx::util::query_counters::start() + 0x1cf in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b71526514  : hpx::components::server::runtime_support::call_startup_functions() + 0x34 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b713c6d59  : ??? + 0x713c6d59 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b71379bc8  : ??? + 0x71379bc8 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b71430da6  : ??? + 0x71430da6 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f4b71431169  : ??? + 0x71431169 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
[what]: unknown performance counter: '/counters/parcelqueue(locality#1/total)/length/instantaneous': HPX(bad_parameter)
[locality]: 1
[function]: query_counters
[file]: /home/wash/hpx/src/util/query_counters.cpp
[line]: 32
[os-thread]: 0
[thread_id]: 00000000021d1018
[thread_name]: hpx::components::server::runtime_support::call_startup_functions_action
[version]: V0.7.0-trunk (AGAS: V2.1), SVN: 5953:5954M
[boost]: V1.47.0
[build-type]: release
[date]: Oct 13 2011 01:27:46
[platform]: linux
[compiler]: GNU C++ version 4.6.1
[stdlib]: GNU libstdc++ version 20110714

Aborted
</:exception>

[02:35:23]:wash:/home/wash/hpx/gcc-4.6.1-release$ bin/sheneos_test --print-counter '/counters/parcelqueue(locality#1/total)/length/instantaneous' --file /shared/hdf5/sheneos_220r_180t_50y_extT_analmu_20100322_SVNr28.h5 
Create interpolator: 0.902185 [s]
Finished task: 0
Running tests: 0.150494 [s]
Received Segmentation fault, 11 frames:
0x7f05702fd365  : ??? + 0x702fd365 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f05702f7fe9  : hpx::detail::backtrace() + 0x9 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f05702c05d6  : hpx::termination_handler(int) + 0x16 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f056e9c3020  : ??? + 0x6e9c3020 in /lib/x86_64-linux-gnu/libpthread.so.0
0x7f057053edcc  : hpx::performance_counters::stubs::performance_counter::get_value(hpx::naming::id_type const&, hpx::error_code&) + 0xc in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f057058dba8  : hpx::util::query_counters::evaluate() + 0x1a8 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f0570477594  : hpx::components::server::runtime_support::call_shutdown_functions(bool) + 0x34 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f05704a002f  : ??? + 0x704a002f in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f057049907b  : ??? + 0x7049907b in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f0570381da6  : ??? + 0x70381da6 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
0x7f0570382169  : ??? + 0x70382169 in /shared/home/wash/hpx/gcc-4.6.1-release/lib/hpx/libhpx.so.0
Aborted

该提问来源于开源项目：STEllAR-GROUP/hpx

Answer 1

[comment by blelbach] [Trac time Thu Oct 13 02:42:08 2011] Debug build reveals an assertion:

[02:44:59]:wash:/home/wash/hpx/gcc-4.6.1-debug$ bin/sheneos_test --file /shared/hdf5/sheneos_220r_180t_50y_extT_analmu_20100322_SVNr28.h5 --list-counters --print-counter '/counters/time(locality#1/total)/idle-rate'
registered performance counters
/counters/agas(locality#1/total)/cache/evictions
/counters/agas(locality#1/total)/cache/hits
/counters/agas(locality#1/total)/cache/insertions
/counters/agas(locality#1/total)/cache/misses
/counters/parcelqueue(locality#1/total)/length/instantaneous
/counters/parcels(locality#1/total)/count/received
/counters/parcels(locality#1/total)/count/sent
/counters/queue(locality#1/os-thread0)/length
/counters/queue(locality#1/total)/length
/counters/runtime(locality#1/total)/uptime
/counters/threads(locality#1/os-thread#0)/count/cumulative/all
/counters/threads(locality#1/os-thread#0)/count/instantaneous/active
/counters/threads(locality#1/os-thread#0)/count/instantaneous/all
/counters/threads(locality#1/os-thread#0)/count/instantaneous/pending
/counters/threads(locality#1/os-thread#0)/count/instantaneous/suspended
/counters/threads(locality#1/os-thread#0)/count/instantaneous/terminated
/counters/threads(locality#1/total)/count/cumulative/all
/counters/threads(locality#1/total)/count/instantaneous/active
/counters/threads(locality#1/total)/count/instantaneous/all
/counters/threads(locality#1/total)/count/instantaneous/pending
/counters/threads(locality#1/total)/count/instantaneous/suspended
/counters/threads(locality#1/total)/count/instantaneous/terminated
/counters/threads(locality#1/total)/count/objects
/counters/time(locality#1/os-thread#0)/idle-rate
/counters/time(locality#1/total)/idle-rate
Create interpolator: 0.912288 [s]
Finished task: 0
Running tests: 0.858677 [s]

[stack_trace]: 29 frames:
0x7fd40987ffe8  : boost::stack_trace::trace(void**, int) + 0x20 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093dd11f  : boost::backtrace::backtrace(unsigned long) + 0x7d in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093dd27a  : boost::trace() + 0x1e in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093db270  : hpx::detail::backtrace() + 0x18 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093dd885  : void hpx::detail::throw_exception<:exception>(hpx::exception const&, std::string const&, std::string const&, int) + 0x6d in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093db4ea  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x23d in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4093db2ad  : hpx::detail::assertion_failed_msg(char const*, char const*, char const*, char const*, long) + 0x0 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x6fb6d0        : boost::asio::io_service::id::id() + 0x0 in bin/sheneos_test
0x7fd4097d9cae  : hpx::util::query_counters::evaluate() + 0x54 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7890c2        : void boost::_mfi::mf0<void hpx::util::query_counters>::call<:shared_ptr> >(boost::shared_ptr<:util::query_counters>&, void const*) const + 0x68 in bin/sheneos_test
0x7863c1        : void boost::_mfi::mf0<void hpx::util::query_counters>::operator()<:shared_ptr> >(boost::shared_ptr<:util::query_counters>&) const + 0x2f in bin/sheneos_test
0x784734        : void boost::_bi::list1<:_bi::value> > >::operator()<:_mfi::mf0 hpx::util::query_counters>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void hpx::util::query_counters>&, boost::_bi::list0&, int) + 0x4a in bin/sheneos_test
0x782285        : boost::_bi::bind_t<void boost::_mfi::mf0 hpx::util::query_counters>, boost::_bi::list1<:_bi::value> > > >::operator()() + 0x39 in bin/sheneos_test
0x77f08e        : boost::detail::function::void_function_obj_invoker0<:_bi::bind_t boost::_mfi::mf0 hpx::util::query_counters>, boost::_bi::list1<:_bi::value> > > >, void>::invoke(boost::detail::function::function_buffer&) + 0x23 in bin/sheneos_test
0x7fd4093887d0  : boost::function0<void>::operator()() const + 0x52 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd40962faf8  : hpx::components::server::runtime_support::call_shutdown_functions(bool) + 0x118 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd409663812  : boost::_mfi::mf1<void hpx::components::server::runtime_support bool>::operator()(hpx::components::server::runtime_support*, bool) const + 0x62 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd409662f49  : void boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> >::operator()<:_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void hpx::components::server::runtime_support bool>&, boost::_bi::list0&, int) + 0x6d in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd409660a35  : boost::_bi::bind_t<void boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >::operator()() + 0x39 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd40965b761  : hpx::threads::thread_state_enum hpx::actions::action<:components::server::runtime_support boost::fusion::unused_type boost::fusion::vector boost::fusion::void_>, hpx::actions::action1<:components::server::runtime_support bool hpx::actions::detail::this_type>, (hpx::threads::thread_priority)0>::continuation_thread_function_void<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > > >(boost::shared_ptr<:actions::continuation>, boost::tuples::tuple<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>) + 0x100 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4096645e8  : hpx::threads::thread_state_enum boost::_bi::list2<:_bi::value> >, boost::_bi::value<:tuples::tuple boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > >::operator()<:threads::thread_state_enum hpx::threads::thread_state_enum>, boost::tuples::tuple<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>), boost::_bi::list1<:threads::thread_state_ex_enum> >(boost::_bi::type<:threads::thread_state_enum>, hpx::threads::thread_state_enum (*&)(boost::shared_ptr<:actions::continuation>, boost::tuples::tuple<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>), boost::_bi::list1<:threads::thread_state_ex_enum>&, long) + 0xba in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd409663f6e  : hpx::threads::thread_state_enum boost::_bi::bind_t<:threads::thread_state_enum hpx::threads::thread_state_enum>, boost::tuples::tuple<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>), boost::_bi::list2<:_bi::value> >, boost::_bi::value<:tuples::tuple boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > >::operator()<:threads::thread_state_ex_enum>(hpx::threads::thread_state_ex_enum&) + 0x44 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd409663899  : boost::detail::function::function_obj_invoker1<:_bi::bind_t hpx::threads::thread_state_enum>, boost::tuples::tuple<:_bi::bind_t boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type>), boost::_bi::list2<:_bi::value> >, boost::_bi::value<:tuples::tuple boost::_mfi::mf1 hpx::components::server::runtime_support bool>, boost::_bi::list2<:_bi::value>, boost::_bi::value<bool> > >, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type, boost::tuples::null_type> > > >, hpx::threads::thread_state_enum, hpx::threads::thread_state_ex_enum>::invoke(boost::detail::function::function_buffer&, hpx::threads::thread_state_ex_enum) + 0x2d in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094b8796  : boost::function1<:threads::thread_state_enum hpx::threads::thread_state_ex_enum>::operator()(hpx::threads::thread_state_ex_enum) const + 0x5a in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094b80b8  : boost::coroutines::detail::unpacker_n<:coroutines::tuple_traits boost::tuples::null_type>, 1>::result<:function>, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type> >::type boost::coroutines::detail::unpacker_n<:coroutines::tuple_traits boost::tuples::null_type>, 1>::operator()<:function>, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type> >(boost::function<:threads::thread_state_enum>&, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type>&) + 0x30 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094b6ffd  : boost::coroutines::detail::unpacker_n<:coroutines::tuple_traits boost::tuples::null_type>, boost::coroutines::tuple_traits<:threads::thread_state_ex_enum boost::tuples::null_type>::length>::result<:function>, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type> >::type boost::coroutines::detail::unpack<:coroutines::tuple_traits boost::tuples::null_type>, boost::function<:threads::thread_state_enum>, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type> >(boost::function<:threads::thread_state_enum>, boost::tuples::tuple<:threads::thread_state_ex_enum boost::tuples::null_type>&, boost::coroutines::detail::trait_tag<:coroutines::tuple_traits boost::tuples::null_type> >) + 0x27 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094b4023  : boost::disable_if<:is_void>, void>::type boost::coroutines::detail::coroutine_impl_wrapper<:function>, boost::coroutines::coroutine<:threads::thread_state_enum hpx::threads::detail::coroutine_allocator boost::coroutines::detail::oslinux64::x86_64_linux_context_impl>, boost::coroutines::detail::oslinux64::x86_64_linux_context_impl, hpx::threads::detail::coroutine_allocator>::do_call<:threads::thread_state_enum>(boost::coroutines::detail::coroutine_impl_wrapper<:function>, boost::coroutines::coroutine<:threads::thread_state_enum hpx::threads::detail::coroutine_allocator boost::coroutines::detail::oslinux64::x86_64_linux_context_impl>, boost::coroutines::detail::oslinux64::x86_64_linux_context_impl, hpx::threads::detail::coroutine_allocator>::dummy<1>) + 0xa1 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094af62f  : boost::coroutines::detail::coroutine_impl_wrapper<:function>, boost::coroutines::coroutine<:threads::thread_state_enum hpx::threads::detail::coroutine_allocator boost::coroutines::detail::oslinux64::x86_64_linux_context_impl>, boost::coroutines::detail::oslinux64::x86_64_linux_context_impl, hpx::threads::detail::coroutine_allocator>::operator()() + 0x5b in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
0x7fd4094a96be  : void boost::coroutines::detail::oslinux64::trampoline<:coroutines::detail::coroutine_impl_wrapper>, boost::coroutines::coroutine<:threads::thread_state_enum hpx::threads::detail::coroutine_allocator boost::coroutines::detail::oslinux64::x86_64_linux_context_impl>, boost::coroutines::detail::oslinux64::x86_64_linux_context_impl, hpx::threads::detail::coroutine_allocator> >(boost::coroutines::detail::coroutine_impl_wrapper<:function>, boost::coroutines::coroutine<:threads::thread_state_enum hpx::threads::detail::coroutine_allocator boost::coroutines::detail::oslinux64::x86_64_linux_context_impl>, boost::coroutines::detail::oslinux64::x86_64_linux_context_impl, hpx::threads::detail::coroutine_allocator>*) + 0x18 in /shared/home/wash/hpx/gcc-4.6.1-debug/lib/hpx/libhpxd.so.0
[what]: assertion 'ids_.size() == names_.size()' failed: HPX(assertion_failure)
[locality]: 1
[function]: void hpx::util::query_counters::evaluate()
[file]: /home/wash/hpx/src/util/query_counters.cpp
[line]: 42
[os-thread]: 0
[thread_id]: 0000000001add628
[thread_name]: hpx::components::server::runtime_support::call_shutdown_functions_action
[version]: V0.7.0-trunk (AGAS: V2.1), SVN: 5959
[boost]: V1.47.0
[build-type]: debug
[date]: Oct 13 2011 02:40:45
[platform]: linux
[compiler]: GNU C++ version 4.6.1
[stdlib]: GNU libstdc++ version 20110714

Aborted
</:threads::thread_state_enum></:function></:threads::thread_state_enum></:coroutines::detail::coroutine_impl_wrapper></:threads::thread_state_enum></:function></:threads::thread_state_enum></:function></:threads::thread_state_enum></:threads::thread_state_enum></:function></:is_void></:coroutines::tuple_traits></:threads::thread_state_ex_enum></:threads::thread_state_enum></:threads::thread_state_ex_enum></:threads::thread_state_enum></:coroutines::tuple_traits></:threads::thread_state_ex_enum></:function></:threads::thread_state_ex_enum></:coroutines::tuple_traits></:threads::thread_state_ex_enum></:threads::thread_state_enum></:threads::thread_state_ex_enum></:function></:coroutines::tuple_traits></:threads::thread_state_ex_enum></:function></:coroutines::tuple_traits></:threads::thread_state_enum></bool></:_bi::value></:tuples::tuple></:_bi::value></bool></:_bi::value></:_bi::bind_t></:_bi::bind_t></:threads::thread_state_ex_enum></bool></:_bi::value></:tuples::tuple></:_bi::value></bool></:_bi::value></:_bi::bind_t></:threads::thread_state_enum></:threads::thread_state_ex_enum></bool></:_bi::value></:_bi::bind_t></:actions::continuation></:threads::thread_state_enum></:threads::thread_state_ex_enum></bool></:_bi::value></:_bi::bind_t></:threads::thread_state_enum></bool></:_bi::value></:tuples::tuple></:_bi::value></bool></:_bi::value></:_bi::bind_t></:actions::continuation></bool></:_bi::value></:_bi::bind_t></:components::server::runtime_support></:components::server::runtime_support></bool></:_bi::value></void></void></void></:_mfi::mf1></bool></:_bi::value></void></void></:_bi::value></:_bi::bind_t></:_bi::value></void></void></void></:_mfi::mf0></:_bi::value></:util::query_counters></:shared_ptr></void></:util::query_counters></:shared_ptr></void></:exception>