ddl: crash in syncSchemaDiffs

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

From the test output:
https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14900/pipeline/368/

[2023-11-01T07:21:21.091Z] [2023/11/01 15:16:06.582 +08:00] [ERROR] [BaseDaemon.cpp:563] ["
       0xc6a9ebb     faultSignalHandler(int, siginfo_t*, void*) [tiflash+208314043]
  0x7f04ff4e9630     <unknown symbol> [libpthread.so.0+63024]
  0x7f04fed28387     gsignal [libc.so.6+222087]
  0x7f04fed29a78     __GI_abort [libc.so.6+227960]
  0x7f050ba5b66f     std::__1::__libcpp_abort_debug_function(std::__1::__libcpp_debug_info const&) [libc++.so.1+464495]
       0xdead496     std::__1::optional<DB::SchemaDiff>::operator->() [tiflash+233493654]
       0xdeac3f5     DB::TiDBSchemaSyncer<false, false>::syncSchemaDiffs(DB::Context&, DB::SchemaGetter&, long) [tiflash+233489397]
       0xdeabe16     DB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+233487894]
       0x4387ac0     DB::TiDBSchemaSyncerManager::syncSchemas(DB::Context&, unsigned int) [tiflash+70810304]
       0xde7a8ec     DB::SchemaSyncService::syncSchemas(unsigned int) [tiflash+233285868]
       0xde815c5     DB::SchemaSyncService::addKeyspaceGCTasks()::$_2::operator()() const [tiflash+233313733]
       0xde814c5     decltype(static_cast<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2&>(fp)()) std::__1::__invoke<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2&>(DB::SchemaSyncService::addKeyspaceGCTasks()::$_2&) [tiflash+233313477]
       0xde8147d     bool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2&>(DB::SchemaSyncService::addKeyspaceGCTasks()::$_2&) [tiflash+233313405]
       0xde8144d     std::__1::__function::__alloc_func<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2, std::__1::allocator<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2>, bool ()>::operator()() [tiflash+233313357]
       0xde80659     std::__1::__function::__func<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2, std::__1::allocator<DB::SchemaSyncService::addKeyspaceGCTasks()::$_2>, bool ()>::operator()() [tiflash+233309785]
       0x69175a2     std::__1::__function::__value_func<bool ()>::operator()() const [tiflash+110196130]
       0x69170f5     std::__1::function<bool ()>::operator()() const [tiflash+110194933]
       0xd5ac502     DB::BackgroundProcessingPool::threadFunction(unsigned long) [tiflash+224052482]
       0xd5ad40c     DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [tiflash+224056332]
       0xd5ad3b5     decltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [tiflash+224056245]
       0xd5ad395     void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [tiflash+224056213]
       0xd5ad132     void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [tiflash+224055602]
  0x7f04ff4e1ea5     start_thread [libpthread.so.0+32421]"] [source=BaseDaemon] [thread_id=733]

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

If the diff == std::nullopt, then it will crash at line 124 /cc @hongyunyan

tiflash/dbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp

Lines 116 to 132 in a55a10f

	std::optional<SchemaDiff> diff = getter.getSchemaDiff(used_version);
	if (used_version == latest_version && !diff)
	{
	--used_version;
	break;
	}
	if (diff->regenerate_schema_map)
	{
	// If `schema_diff.regenerate_schema_map` == true, return `-1` directly, let TiFlash reload schema info from TiKV.
	LOG_INFO(log, "Meets a schema diff with regenerate_schema_map flag");
	return -1;
	}
	SchemaBuilder<Getter, NameMapper> builder(getter, context, databases, table_id_map, shared_mutex_for_databases);
	builder.applyDiff(*diff);

[JaySon-Huang]

similari issue in another test

https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/14900/pipeline/355/

[2023-11-01T07:59:45.827Z] [2023/11/01 15:07:58.095 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=749]
[2023-11-01T07:59:45.827Z] [2023/11/01 15:07:58.095 +08:00] [ERROR] [BaseDaemon.cpp:371] ["(from thread 280) Received signal Aborted(6)."] [source=BaseDaemon] [thread_id=749]
[2023-11-01T07:59:45.827Z] [2023/11/01 15:07:58.116 +08:00] [ERROR] [BaseDaemon.cpp:563] ["\n       0xc6a9ebb\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+208314043]\n  0x7f88474e9630\t<unknown symbol> [libpthread.so.0+63024]\n  0x7f8846d28387\tgsignal [libc.so.6+222087]\n  0x7f8846d29a78\t__GI_abort [libc.so.6+227960]\n  0x7f88539b366f\tstd::__1::__libcpp_abort_debug_function(std::__1::__libcpp_debug_info const&) [libc++.so.1+464495]\n       0xdead496\tstd::__1::optional<DB::SchemaDiff>::operator->() [tiflash+233493654]\n       0xdeac3f5\tDB::TiDBSchemaSyncer<false, false>::syncSchemaDiffs(DB::Context&, DB::SchemaGetter&, long) [tiflash+233489397]\n       0xdeabe16\tDB::TiDBSchemaSyncer<false, false>::syncSchemas(DB::Context&) [tiflash+233487894]\n       0x4387ac0\tDB::TiDBSchemaSyncerManager::syncSchemas(DB::Context&, unsigned int) [tiflash+70810304]\n       0xc9a1d7f\tDB::dbgFuncRefreshSchemas(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) [tiflash+211426687]\n       0xc9466d9\tdecltype(static_cast<void (*&>(fp)(static_cast<DB::Context&>(fp0), static_cast<std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&>(fp0), static_cast<std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>>(fp0))) std::__1::__invoke<void (*&)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)> >(void (*&)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>&&) [tiflash+211052249]\n       0xc94664c\tvoid std::__1::__invoke_void_return_wrapper<void, true>::__call<void (*&)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)> >(void (*&)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>&&) [tiflash+211052108]\n       0xc9465bc\tstd::__1::__function::__alloc_func<void (*)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), std::__1::allocator<void (*)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>, void (DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>::operator()(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>&&) [tiflash+211051964]\n       0xc9456bb\tstd::__1::__function::__func<void (*)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>), std::__1::allocator<void (*)(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>, void (DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>::operator()(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>&&) [tiflash+211048123]\n       0xc9496d4\tstd::__1::__function::__value_func<void (DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>::operator()(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>&&) const [tiflash+211064532]\n       0xc940c07\tstd::__1::function<void (DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)>::operator()(DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) const [tiflash+211028999]\n       0xc93c81e\tDB::DBGInvoker::invokeSchemaless(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<void (DB::Context&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>)> const&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&) [tiflash+211011614]\n       0xc93c384\tDB::DBGInvoker::invoke(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::shared_ptr<DB::IAST>, std::__1::allocator<std::__1::shared_ptr<DB::IAST> > > const&) [tiflash+211010436]\n       0xd3db5af\tDB::InterpreterDBGInvokeQuery::execute() [tiflash+222148015]\n       0xd39a4ef\tDB::(anonymous namespace)::executeQueryImpl(DB::SQLQuerySource&, DB::Context&, bool, DB::QueryProcessingStage::Enum) [tiflash+221881583]\n       0xd399e73\tDB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum) [tiflash+221879923]\n       0x446cea5\tDB::TCPHandler::runImpl() [tiflash+71749285]\n       0x4471c25\tDB::TCPHandler::run() [tiflash+71769125]\n       0xf74ddb9\tPoco::Net::TCPServerConnection::start() [tiflash+259317177]\n       0xf74e54a\tPoco::Net::TCPServerDispatcher::run() [tiflash+259319114]\n       0xf917324\tPoco::PooledThread::run() [tiflash+261190436]\n       0xf91333a\tPoco::(anonymous namespace)::RunnableHolder::run() [tiflash+261174074]\n       0xf9123f8\tPoco::ThreadImpl::runnableEntry(void*) [tiflash+261170168]\n       0xf914b40\tdecltype(static_cast<void* (*>(fp)(static_cast<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) [tiflash+261180224]\n       0xf914b0e\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) [tiflash+261180174]\n       0xf914812\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) [tiflash+261179410]\n  0x7f88474e1ea5\tstart_thread [libpthread.so.0+32421]"] [source=BaseDaemon] [thread_id=749]

[JaySon-Huang]

Duplicated with #8578, fixed by #8623