gitlab备份恢复数据后管理员保存参数500报错

Question

问题遇到的现象和发生背景

docker gitlab 使用备份恢复数据后管理员设置参数500报错
如

点击保存报错

还有些日志报错

root@gitlab:/# gitlab-ctl tail|grep error
==> /var/log/gitlab/nginx/gitlab_error.log <==
==> /var/log/gitlab/nginx/error.log <==
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2024-04-29T06:19:22Z"}
{"severity":"WARN","time":"2024-04-29T06:19:11.969Z","message":"Job arguments to Projects::GitGarbageCollectWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby calling `Sidekiq.strict_args!` during Sidekiq initialization.\n"}
{"catfile.duration_ms":0,"catfile.flush_count":1,"catfile.flush_ms":0,"catfile.read_object_count":1,"catfile.read_object_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":2,"command.cpu_time_ms":2,"command.inblock":0,"command.majflt":0,"command.maxrss":305528,"command.minflt":916,"command.oublock":0,"command.real_time_ms":5,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":2,"command.user_time_ms":0,"component":"gitaly.StreamServerInterceptor","correlation_id":"01HWM83REH676NWPFWZ3SWTN4R","error":"tree entry not found","error_metadata":{"path":".gitlab-ci.yml"},"grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.method_operation":"accessor","grpc.meta.method_scope":"repository","grpc.meta.method_type":"server_stream","grpc.method":"TreeEntry","grpc.request.deadline":"2024-04-29T06:19:41.623","grpc.request.fullMethod":"/gitaly.CommitService/TreeEntry","grpc.request.glProjectPath":"beauty/beauty-front-b-pc","grpc.request.glRepository":"project-44","grpc.request.payload_bytes":197,"grpc.request.repoPath":"@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.CommitService","grpc.start_time":"2024-04-29T06:19:11.923","grpc.time_ms":3.661,"level":"info","msg":"finished streaming call with code NotFound","pid":19290,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2024-04-29T06:19:11.927Z","user_id":"4","username":"liyang"}
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
{"severity":"WARN","time":"2024-04-29T03:14:23.304Z","message":"Job arguments to Users::TrackNamespaceVisitsWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today\nby calling `Sidekiq.strict_args!` during Sidekiq initialization.\n"}
2024-04-29 04:29:13 +0000 HTTP parse error, malformed request ("GET /cgi-bin/luci/;stok=/locale" - (127.0.0.1)): #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
{"severity":"INFO","time":"2024-04-29T06:20:03.164Z","retry":0,"queue":"default","version":0,"queue_namespace":"cronjob","args":[],"class":"Packages::CleanupPackageRegistryWorker","jid":"27d35d82c71de0909d6b9cf9","created_at":"2024-04-29T06:20:02.974Z","meta.caller_id":"Cronjob","correlation_id":"1c188d7210a0f68031abb88871bb334e","meta.root_caller_id":"Cronjob","meta.feature_category":"package_registry","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:31163b2078644913f89feb2245c5fa62ac9118d29e04cf14ce0b5def57bba8ea","size_limiter":"validated","enqueued_at":"2024-04-29T06:20:02.978Z","job_size_bytes":2,"pid":19420,"message":"Packages::CleanupPackageRegistryWorker JID-27d35d82c71de0909d6b9cf9: done: 0.181879 sec","job_status":"done","scheduling_latency_s":0.003915,"redis_calls":4,"redis_duration_s":0.016692,"redis_read_bytes":4,"redis_write_bytes":574,"redis_feature_flag_calls":2,"redis_feature_flag_duration_s":0.00417,"redis_feature_flag_read_bytes":2,"redis_feature_flag_write_bytes":388,"redis_queues_calls":1,"redis_queues_duration_s":0.000151,"redis_queues_read_bytes":1,"redis_queues_write_bytes":63,"redis_queues_metadata_calls":1,"redis_queues_metadata_duration_s":0.012371,"redis_queues_metadata_read_bytes":1,"redis_queues_metadata_write_bytes":123,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":7,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.135,"db_main_duration_s":0.135,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.018144,"mem_objects":5054,"mem_bytes":348936,"mem_mallocs":2673,"mem_total_bytes":551096,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.packages_cleanup_package_registry_worker.pending_destruction_package_files_count":0,"extra.packages_cleanup_package_registry_worker.processing_package_files_count":0,"extra.packages_cleanup_package_registry_worker.error_package_files_count":0,"extra.packages_cleanup_package_registry_worker.pending_cleanup_policies_count":0,"duration_s":0.181879,"completed_at":"2024-04-29T06:20:03.164Z","load_balancing_strategy":"primary","db_duration_s":0.13443,"urgency":"low","target_duration_s":300,"target_scheduling_latency_s":60}

操作环境、软件版本等信息

环境 CentOS Linux release 7.9.2009 (Core)
软件版本 docker-ce-26.1.0-1.el7
gitlab-ce:16.5.7-ce.0

尝试过的解决方法

故障出现后运行 gitlab-rake gitlab:doctor:secrets

root@gitlab:/# gitlab-rake gitlab:doctor:secrets
I, [2024-04-29T05:54:01.062312 #18019]  INFO -- : Checking encrypted values in the database
I, [2024-04-29T05:54:14.487847 #18019]  INFO -- : - Ci::InstanceVariable failures: 0
I, [2024-04-29T05:54:14.497254 #18019]  INFO -- : - Ci::PipelineVariable failures: 0
I, [2024-04-29T05:54:14.505169 #18019]  INFO -- : - Ci::PipelineScheduleVariable failures: 0
I, [2024-04-29T05:54:14.513813 #18019]  INFO -- : - Ci::Variable failures: 0
I, [2024-04-29T05:54:14.521710 #18019]  INFO -- : - Ci::GroupVariable failures: 0
I, [2024-04-29T05:54:14.530938 #18019]  INFO -- : - Ci::Trigger failures: 0
I, [2024-04-29T05:54:14.540568 #18019]  INFO -- : - Ci::JobVariable failures: 0
I, [2024-04-29T05:54:14.542682 #18019]  INFO -- : - Gitlab::BackgroundMigration::EncryptCiTriggerToken::CiTrigger failures: 0
I, [2024-04-29T05:54:15.928769 #18019]  INFO -- : - ApplicationSetting failures: 1
I, [2024-04-29T05:54:16.009942 #18019]  INFO -- : - User failures: 0
I, [2024-04-29T05:54:16.019489 #18019]  INFO -- : - Clusters::Platforms::Kubernetes failures: 0
I, [2024-04-29T05:54:16.044869 #18019]  INFO -- : - Snippet failures: 0
I, [2024-04-29T05:54:16.047200 #18019]  INFO -- : - PersonalSnippet failures: 0
I, [2024-04-29T05:54:16.049512 #18019]  INFO -- : - ProjectSnippet failures: 0
I, [2024-04-29T05:54:16.057101 #18019]  INFO -- : - ProjectImportData failures: 0
I, [2024-04-29T05:54:16.083591 #18019]  INFO -- : - Integration failures: 0
I, [2024-04-29T05:54:16.086714 #18019]  INFO -- : - Integrations::AppleAppStore failures: 0
I, [2024-04-29T05:54:16.089412 #18019]  INFO -- : - Integrations::Asana failures: 0
I, [2024-04-29T05:54:16.092161 #18019]  INFO -- : - Integrations::Assembla failures: 0
I, [2024-04-29T05:54:16.095001 #18019]  INFO -- : - Integrations::BaseCi failures: 0
I, [2024-04-29T05:54:16.097213 #18019]  INFO -- : - Integrations::Bamboo failures: 0
I, [2024-04-29T05:54:16.099725 #18019]  INFO -- : - Integrations::Buildkite failures: 0
I, [2024-04-29T05:54:16.102186 #18019]  INFO -- : - Integrations::DroneCi failures: 0
I, [2024-04-29T05:54:16.104598 #18019]  INFO -- : - Integrations::Jenkins failures: 0
I, [2024-04-29T05:54:16.107129 #18019]  INFO -- : - Integrations::Teamcity failures: 0
I, [2024-04-29T05:54:16.109645 #18019]  INFO -- : - Integrations::MockCi failures: 0
I, [2024-04-29T05:54:16.112225 #18019]  INFO -- : - Integrations::BaseIssueTracker failures: 0
I, [2024-04-29T05:54:16.114714 #18019]  INFO -- : - Integrations::Bugzilla failures: 0
I, [2024-04-29T05:54:16.117289 #18019]  INFO -- : - Integrations::Clickup failures: 0
I, [2024-04-29T05:54:16.119615 #18019]  INFO -- : - Integrations::CustomIssueTracker failures: 0
I, [2024-04-29T05:54:16.122114 #18019]  INFO -- : - Integrations::Ewm failures: 0
I, [2024-04-29T05:54:16.124645 #18019]  INFO -- : - Integrations::Jira failures: 0
I, [2024-04-29T05:54:16.126906 #18019]  INFO -- : - Integrations::Redmine failures: 0
I, [2024-04-29T05:54:16.129420 #18019]  INFO -- : - Integrations::Youtrack failures: 0
I, [2024-04-29T05:54:16.131670 #18019]  INFO -- : - Integrations::Zentao failures: 0
I, [2024-04-29T05:54:16.134079 #18019]  INFO -- : - Integrations::Campfire failures: 0
I, [2024-04-29T05:54:16.136684 #18019]  INFO -- : - Integrations::BaseThirdPartyWiki failures: 0
I, [2024-04-29T05:54:16.138850 #18019]  INFO -- : - Integrations::Confluence failures: 0
I, [2024-04-29T05:54:16.141239 #18019]  INFO -- : - Integrations::Shimo failures: 0
I, [2024-04-29T05:54:16.143744 #18019]  INFO -- : - Integrations::Datadog failures: 0
I, [2024-04-29T05:54:16.146430 #18019]  INFO -- : - Integrations::BaseChatNotification failures: 0
I, [2024-04-29T05:54:16.148937 #18019]  INFO -- : - Integrations::Discord failures: 0
I, [2024-04-29T05:54:16.151415 #18019]  INFO -- : - Integrations::HangoutsChat failures: 0
I, [2024-04-29T05:54:16.153674 #18019]  INFO -- : - Integrations::Mattermost failures: 0
I, [2024-04-29T05:54:16.156049 #18019]  INFO -- : - Integrations::MicrosoftTeams failures: 0
I, [2024-04-29T05:54:16.158483 #18019]  INFO -- : - Integrations::Pumble failures: 0
I, [2024-04-29T05:54:16.160799 #18019]  INFO -- : - Integrations::BaseSlackNotification failures: 0
I, [2024-04-29T05:54:16.163331 #18019]  INFO -- : - Integrations::Slack failures: 0
I, [2024-04-29T05:54:16.165899 #18019]  INFO -- : - Integrations::GitlabSlackApplication failures: 0
I, [2024-04-29T05:54:16.168242 #18019]  INFO -- : - Integrations::Telegram failures: 0
I, [2024-04-29T05:54:16.170773 #18019]  INFO -- : - Integrations::UnifyCircuit failures: 0
I, [2024-04-29T05:54:16.173128 #18019]  INFO -- : - Integrations::WebexTeams failures: 0
I, [2024-04-29T05:54:16.175574 #18019]  INFO -- : - Integrations::EmailsOnPush failures: 0
I, [2024-04-29T05:54:16.178064 #18019]  INFO -- : - Integrations::ExternalWiki failures: 0
I, [2024-04-29T05:54:16.180346 #18019]  INFO -- : - Integrations::GooglePlay failures: 0
I, [2024-04-29T05:54:16.182887 #18019]  INFO -- : - Integrations::Harbor failures: 0
I, [2024-04-29T05:54:16.185317 #18019]  INFO -- : - Integrations::Irker failures: 0
I, [2024-04-29T05:54:16.187748 #18019]  INFO -- : - Integrations::BaseSlashCommands failures: 0
I, [2024-04-29T05:54:16.190197 #18019]  INFO -- : - Integrations::MattermostSlashCommands failures: 0
I, [2024-04-29T05:54:16.192701 #18019]  INFO -- : - Integrations::SlackSlashCommands failures: 0
I, [2024-04-29T05:54:16.194917 #18019]  INFO -- : - Integrations::Packagist failures: 0
I, [2024-04-29T05:54:16.197370 #18019]  INFO -- : - Integrations::PipelinesEmail failures: 0
I, [2024-04-29T05:54:16.199609 #18019]  INFO -- : - Integrations::Pivotaltracker failures: 0
I, [2024-04-29T05:54:16.202085 #18019]  INFO -- : - Integrations::BaseMonitoring failures: 0
I, [2024-04-29T05:54:16.204582 #18019]  INFO -- : - Integrations::Prometheus failures: 0
I, [2024-04-29T05:54:16.206781 #18019]  INFO -- : - Integrations::MockMonitoring failures: 0
I, [2024-04-29T05:54:16.209252 #18019]  INFO -- : - Integrations::Pushover failures: 0
I, [2024-04-29T05:54:16.211662 #18019]  INFO -- : - Integrations::SquashTm failures: 0
I, [2024-04-29T05:54:16.222596 #18019]  INFO -- : - AlertManagement::HttpIntegration failures: 0
I, [2024-04-29T05:54:16.231099 #18019]  INFO -- : - ChatName failures: 0
I, [2024-04-29T05:54:16.239895 #18019]  INFO -- : - GrafanaIntegration failures: 0
I, [2024-04-29T05:54:16.250576 #18019]  INFO -- : - JiraConnectInstallation failures: 0
I, [2024-04-29T05:54:16.273202 #18019]  INFO -- : - PagesDomain failures: 0
I, [2024-04-29T05:54:16.279914 #18019]  INFO -- : - PagesDomainAcmeOrder failures: 0
I, [2024-04-29T05:54:16.304508 #18019]  INFO -- : - ProjectSetting failures: 0
I, [2024-04-29T05:54:16.308939 #18019]  INFO -- : - RemoteMirror failures: 0
I, [2024-04-29T05:54:16.316598 #18019]  INFO -- : - SlackIntegration failures: 0
I, [2024-04-29T05:54:16.321583 #18019]  INFO -- : - Ai::ServiceAccessToken failures: 0
I, [2024-04-29T05:54:16.326403 #18019]  INFO -- : - Alerting::ProjectAlertingSetting failures: 0
I, [2024-04-29T05:54:16.332389 #18019]  INFO -- : - Atlassian::Identity failures: 0
I, [2024-04-29T05:54:16.338188 #18019]  INFO -- : - BulkImports::Configuration failures: 0
I, [2024-04-29T05:54:16.365435 #18019]  INFO -- : - Clusters::KubernetesNamespace failures: 0
I, [2024-04-29T05:54:16.371756 #18019]  INFO -- : - ErrorTracking::ProjectErrorTrackingSetting failures: 0
I, [2024-04-29T05:54:16.378141 #18019]  INFO -- : - IncidentManagement::ProjectIncidentManagementSetting failures: 0
I, [2024-04-29T05:54:16.384594 #18019]  INFO -- : - Integrations::IssueTrackerData failures: 0
I, [2024-04-29T05:54:16.391778 #18019]  INFO -- : - Integrations::JiraTrackerData failures: 0
I, [2024-04-29T05:54:16.398288 #18019]  INFO -- : - Integrations::ZentaoTrackerData failures: 0
I, [2024-04-29T05:54:16.404066 #18019]  INFO -- : - ServiceDesk::CustomEmailCredential failures: 0
I, [2024-04-29T05:54:16.408785 #18019]  INFO -- : - ServiceDesk::CustomEmailVerification failures: 0
I, [2024-04-29T05:54:16.412096 #18019]  INFO -- : - Clusters::Integrations::Prometheus failures: 0
I, [2024-04-29T05:54:16.419496 #18019]  INFO -- : - Clusters::Providers::Aws failures: 0
I, [2024-04-29T05:54:16.426680 #18019]  INFO -- : - Clusters::Providers::Gcp failures: 0
I, [2024-04-29T05:54:16.433062 #18019]  INFO -- : - Packages::Debian::GroupDistributionKey failures: 0
I, [2024-04-29T05:54:16.439223 #18019]  INFO -- : - Packages::Debian::ProjectDistributionKey failures: 0
I, [2024-04-29T05:54:16.462979 #18019]  INFO -- : - WebHook failures: 0
I, [2024-04-29T05:54:16.465923 #18019]  INFO -- : - ProjectHook failures: 0
I, [2024-04-29T05:54:16.468886 #18019]  INFO -- : - ServiceHook failures: 0
I, [2024-04-29T05:54:16.471694 #18019]  INFO -- : - SystemHook failures: 0
I, [2024-04-29T05:54:16.475244 #18019]  INFO -- : - Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration failures: 0
I, [2024-04-29T05:54:16.518855 #18019]  INFO -- : - Ci::Runner failures: 0
I, [2024-04-29T05:54:16.601593 #18019]  INFO -- : - Ci::Build failures: 0
I, [2024-04-29T05:54:16.683449 #18019]  INFO -- : - Group failures: 0
I, [2024-04-29T05:54:16.691622 #18019]  INFO -- : - DeployToken failures: 0
I, [2024-04-29T05:54:16.767687 #18019]  INFO -- : - Project failures: 0
I, [2024-04-29T05:54:16.778429 #18019]  INFO -- : - Clusters::AgentToken failures: 0
I, [2024-04-29T05:54:16.788415 #18019]  INFO -- : - Operations::FeatureFlagsClient failures: 1
I, [2024-04-29T05:54:16.788496 #18019]  INFO -- : Total: 2 row(s) affected
I, [2024-04-29T05:54:16.788512 #18019]  INFO -- : Done!

按网上找过类似问题解决方案处理，没有任何效果。
如

https://blog.csdn.net/weixin_44295157/article/details/119618816

我想要达到的结果

请哪位大虾给看看怎么处理，怎么才能解决此问题，感谢先！

Answer 1

引自免费微信小程序：皆我百晓生

根据你提供的信息，看起来你在尝试恢复GitLab数据后遇到了500错误，这通常表示服务器内部错误。首先，我们需要确定问题出在哪里。从日志来看，有几个可能的原因：

1. 错误日志中提到了一个关于JSON序列化的警告，虽然它不会直接导致500错误，但最好还是修复一下。
2. 关于gitaly.StreamServerInterceptor的日志表明可能存在树项未找到的问题，这可能是由于数据恢复过程中的错误，或者某个文件在恢复过程中丢失了。
3. 最后，gitlab:doctor:secrets检查结果显示有一个Operations::FeatureFlagsClient失败，这可能与某些功能的启用有关。

针对这些问题，我们可以尝试以下步骤来解决问题：

第一步：修复JSON序列化警告

在你的Sidekiq配置中添加Sidekiq.strict_args!，以确保所有的作业都能正确地序列化到JSON。打开config/sidekiq.yml，并在concurrency:行下方添加以下行：

# config/sidekiq.yml
---
concurrency: 25
dead_max: 8
dead_timeout: 8
max_job_retry_delay: 1.day
strict_args: true

然后重启Sidekiq服务：

sudo gitlab-ctl restart sidekiq

第二步：检查GitLab实例的健康状况

运行gitlab-rake gitlab:check命令以全面检查GitLab实例的健康状况：

sudo gitlab-rake gitlab:check

如果检查结果有异常，请按照提示进行相应的修复。

第三步：检查数据库完整性

执行数据库一致性检查：

sudo gitlab-rake db:migrate:refresh && sudo gitlab-rake db:verify

第四步：检查和修复GitLab CI/CD配置

由于日志中提到tree entry not found，可能涉及到某个项目的CI/CD配置文件（.gitlab-ci.yml）。检查受影响的项目是否存在这个文件，并确认其格式是否正确。

第五步：清理缓存和重启GitLab

清除GitLab的工作目录缓存：

sudo rm -rf /var/opt/gitlab/git-data/repositories/*/tmp/*

然后重启GitLab容器，以便应用所有更改：

sudo gitlab-ctl restart

第六步：检查和更新GitLab配置

确保你的GitLab配置文件没有语法错误，并且所有必要的配置都已正确设置。特别是，检查gitlab.rb或gitlab-secrets.json中的任何加密值是否正确解密。

第七步：进一步排查

如果以上步骤都无法解决问题，你可能需要深入研究具体的错误，比如检查相关的日志文件，或者查看是否有其他可能导致500错误的服务，例如Nginx、Redis等。

如果问题仍然存在，建议联系GitLab支持团队，他们可能能提供更专业的指导，因为这个问题可能涉及到特定的配置或数据损坏。

注意，如果你在操作过程中有任何疑问，或者对上述步骤不熟悉，请寻求具有适当权限的技术人员协助。