Всем привет. Я хочу рассказать о том, как проходило восстановление Sentry после неудачного обновления.
Что же такое Sentry?
Что же такое Sentry?
Это система полного отслеживания ошибок с открытым исходным кодом, которая поддерживает широкий спектр серверных, браузерных, настольных и родных мобильных языков и сред, включая PHP, Node. js, Python, Ruby, C #, Java, Go, React, Angular, Vue, JavaScript и другие.
Немного о том, как просто и непринуждённо всё поломалось.
Жили мы на версии 9.0.0 и пришло время обновиться. Пощупав web интерфейс 10.0.0, принял решение обновиться на неё. Это была самая большая ошибка!
Обновление прошло в штатном режиме. Сначала переход на 9.1.2 затем, на 10.0.0 (иначе нельзя). Далее обнаружил, что не запускается worker. Причиной этому добавление новых приложений и зависимостей.
Попытки собрать приложения и засунуть их в k8s не увенчались успехом.
Было принято решение откатиться назад. Но тут меня ждало разочарование. После миграции на 10 версию, часть данных переезжает в clickhouse и удаляется из Postgres.
После 3 часов работ и привлечения DBA эксперта данные были восстановлены до рабочего состояния.
Как?
Сделали рядом инстанс PG, закинули туда схему от версии 9.0.0 и начали потабличное восстановление. Конечно, без ошибок и напильника тут не обошлось.
На этом история не заканчивается. Пришло время всё же обновиться на версию 9.1.2.
Собрал образ, запускаю:
sentry upgradeи видим вот такое чудесное сообщение:
! I'm not trusting myself; either fix this yourself by fiddling ! with the south_migrationhistory table, or pass --delete-ghost-migrations ! to South to have it delete ALL of these records (this may not be good). Exception in thread Thread-1 (most likely raised during interpreter shutdown)После долгих поисков, проб и ошибок было найдено решение.
Необходимо восстановить схему данных текущей версии.
Для этого клонируем репозиторий:
git clone https://github.com/getsentry/onpremise.gitПереходим в ветку 9.1.2, так как ветки 9.0.0 уже нет.
cd onpremise && git checkout tags/9.1.2Далее правим образ в Dockerfile:
#cat Dockerfile ARG SENTRY_IMAGE FROM ${SENTRY_IMAGE:-sentry:9.0.0}-onbuildЗатем правим compose file
docker-compose.yml
# NOTE: This docker-compose.yml is meant to be just an example of how # you could accomplish this on your own. It is not intended to work in # all use-cases and must be adapted to fit your needs. This is merely # a guideline. # See docs.getsentry.com/on-premise/server/ for full # instructions version: '3.4' x-defaults: &defaults restart: unless-stopped build: context: . depends_on: - redis # - postgres - memcached - smtp env_file: .env environment: SENTRY_MEMCACHED_HOST: memcached SENTRY_REDIS_HOST: redis SENTRY_POSTGRES_HOST: 'sentry.cl.ats' SENTRY_DB_USER: 'sentryDB' SENTRY_DB_NAME: 'sentry_user' SENTRY_DB_PASSWORD: 'sentry_passwd' SENTRY_POSTGRES_PORT: 5432 SENTRY_EMAIL_HOST: smtp volumes: - sentry-data:/var/lib/sentry/files services: smtp: restart: unless-stopped image: tianon/exim4 memcached: restart: unless-stopped image: memcached:1.5-alpine redis: restart: unless-stopped image: redis:3.2-alpine #postgres: # restart: unless-stopped # image: postgres:9.5 # environment: # POSTGRES_HOST_AUTH_METHOD: 'trust' # volumes: # - sentry-postgres:/var/lib/postgresql/data web: <<: *defaults ports: - '9000:9000' cron: <<: *defaults command: run cron worker: <<: *defaults command: run worker volumes: sentry-data: external: true #sentry-postgres: # external: true
Немного пояснений. Так как у меня есть отдельный инстанс базы данных, то создание через compose пропускаю, также не нужен volume sentry-postgres.
После данных манипуляций запускаем сборку:
./install.sh В процессе может выскочить ошибка, о которой я упомянул выше. Не обращаем внимания и после того, как процесс закончится, запускаем следующую команду:
docker-compose run --rm web sentry django migrate --delete-ghost-migrations
Примерный вывод
Creating network "onpremise_default" with the default driver Creating onpremise_memcached_1 ... done Creating onpremise_smtp_1 ... done Creating onpremise_redis_1 ... done 21:30:26 [INFO] sentry.plugins.github: apps-not-configured Running migrations for sentry: - Nothing to migrate. - Loading initial data for sentry. Installed 0 object(s) from 0 fixture(s) Running migrations for nodestore: - Nothing to migrate. - Loading initial data for nodestore. Installed 0 object(s) from 0 fixture(s) Running migrations for search: - Nothing to migrate. - Loading initial data for search. Installed 0 object(s) from 0 fixture(s) Running migrations for social_auth: - Nothing to migrate. - Loading initial data for social_auth. Installed 0 object(s) from 0 fixture(s) Running migrations for tagstore: - Nothing to migrate. - Loading initial data for tagstore. Installed 0 object(s) from 0 fixture(s) Running migrations for jira_ac: - Nothing to migrate. - Loading initial data for jira_ac. Installed 0 object(s) from 0 fixture(s) Running migrations for hipchat_ac: - Nothing to migrate. - Loading initial data for hipchat_ac. Installed 0 object(s) from 0 fixture(s)
Далее можно запускать upgrade.
docker-compose run --rm web sentry upgrade
Stdout
Starting onpremise_smtp_1 ... done Starting onpremise_memcached_1 ... done Starting onpremise_redis_1 ... done 21:30:48 [INFO] sentry.plugins.github: apps-not-configured Syncing... Creating tables ... Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.messages > django.contrib.sessions > django.contrib.sites > django.contrib.staticfiles > crispy_forms > debug_toolbar > raven.contrib.django.raven_compat > rest_framework > sentry.plugins.sentry_interface_types > sentry.plugins.sentry_mail > sentry.plugins.sentry_urls > sentry.plugins.sentry_useragents > sentry.plugins.sentry_webhooks > sudo > south > sentry_plugins.slack Not synced (use migrations): - sentry - sentry.nodestore - sentry.search - social_auth - sentry.tagstore - sentry_plugins.jira_ac - sentry_plugins.hipchat_ac (use ./manage.py migrate to migrate these) Running migrations for sentry: - Nothing to migrate. - Loading initial data for sentry. Installed 0 object(s) from 0 fixture(s) Running migrations for nodestore: - Nothing to migrate. - Loading initial data for nodestore. Installed 0 object(s) from 0 fixture(s) Running migrations for search: - Nothing to migrate. - Loading initial data for search. Installed 0 object(s) from 0 fixture(s) Running migrations for social_auth: - Nothing to migrate. - Loading initial data for social_auth. Installed 0 object(s) from 0 fixture(s) Running migrations for tagstore: - Nothing to migrate. - Loading initial data for tagstore. Installed 0 object(s) from 0 fixture(s) Running migrations for jira_ac: - Nothing to migrate. - Loading initial data for jira_ac. Installed 0 object(s) from 0 fixture(s) Running migrations for hipchat_ac: - Nothing to migrate. - Loading initial data for hipchat_ac. Installed 0 object(s) from 0 fixture(s) Creating missing DSNs Correcting Group.num_comments counter
Итак, схема восстановлена. Можно обновиться на 9.1.2.
Для этого в Dockerfile меняем версию на 9.1.2:
ARG SENTRY_IMAGE FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuildИ запускаем install.sh
./install.shЕсли всё сделано верно, то процесс завершится без ошибок. Вывод должен быть примерно такой:
Stdout
# ./install.sh Checking minimum requirements... Creating volumes for persistent storage... Created sentry-data. .env already exists, skipped creation. Building and tagging Docker images... smtp uses an image, skipping memcached uses an image, skipping redis uses an image, skipping Building web Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Running in 6c97f9fcaf63 DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Removing intermediate container 6c97f9fcaf63 ---> Running in 9e0f65ee3af6 Removing intermediate container 9e0f65ee3af6 ---> Running in 09754c44319c Removing intermediate container 09754c44319c ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_web:latest Building cron Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_cron:latest Building worker Step 1/2 : ARG SENTRY_IMAGE Step 2/2 : FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild # Executing 4 build triggers ---> Using cache ---> Using cache ---> Using cache ---> Using cache ---> a12fa31c2675 Successfully built a12fa31c2675 Successfully tagged onpremise_worker:latest Docker images built. Generating secret key... Secret key written to .env Setting up database... Starting onpremise_smtp_1 ... done Starting onpremise_redis_1 ... done Starting onpremise_memcached_1 ... done 21:35:07 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured. 21:35:10 [INFO] sentry.plugins.github: apps-not-configured Syncing... Creating tables ... Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) Migrating... Running migrations for sentry: - Migrating forwards to 0472_auto__add_field_sentryapp_author. > sentry:0424_auto__add_field_integration_status > sentry:0425_auto__add_index_pullrequest_organization_id_merge_commit_sha > sentry:0425_remove_invalid_github_idps > sentry:0426_auto__add_sentryappinstallation__add_sentryapp__add_field_user_is_sent > sentry:0427_auto__add_eventattachment__add_unique_eventattachment_project_id_event > sentry:0428_auto__add_index_eventattachment_project_id_date_added > sentry:0429_auto__add_integrationexternalproject__add_unique_integrationexternalpr > sentry:0430_auto__add_field_organizationintegration_status > sentry:0431_auto__add_field_externalissue_metadata > sentry:0432_auto__add_field_relay_is_internal > sentry:0432_auto__add_index_userreport_date_added__add_index_eventattachment_date_ > sentry:0433_auto__add_field_relay_is_internal__add_field_userip_country_code__add_ > sentry:0434_auto__add_discoversavedqueryproject__add_unique_discoversavedqueryproj > sentry:0435_auto__add_field_discoversavedquery_created_by > sentry:0436_rename_projectdsymfile_to_projectdebugfile > sentry:0437_auto__add_field_sentryapp_status > sentry:0438_auto__add_index_sentryapp_status__chg_field_sentryapp_proxy_user__chg_ > sentry:0439_auto__chg_field_sentryapp_owner > sentry:0440_auto__del_unique_projectdebugfile_project_debug_id__add_index_projectd > sentry:0441_auto__add_field_projectdebugfile_data > sentry:0442_auto__add_projectcficachefile__add_unique_projectcficachefile_project_ > sentry:0443_auto__add_field_organizationmember_token_expires_at > sentry:0443_auto__del_dsymapp__del_unique_dsymapp_project_platform_app_id__del_ver > sentry:0444_auto__add_sentryappavatar__add_field_sentryapp_redirect_url__add_field > sentry:0445_auto__add_promptsactivity__add_unique_promptsactivity_user_feature_org > sentry:0446_auto__add_index_promptsactivity_project_id > sentry:0447_auto__del_field_promptsactivity_organization__add_field_promptsactivit > sentry:0448_auto__add_field_sentryapp_is_alertable > sentry:0449_auto__chg_field_release_owner > sentry:0450_auto__del_grouphashtombstone__del_unique_grouphashtombstone_project_ha > sentry:0451_auto__del_field_projectbookmark_project_id__add_field_projectbookmark_ > sentry:0452_auto__add_field_sentryapp_events > sentry:0452_auto__del_field_releaseenvironment_organization_id__del_field_releasee > sentry:0453_auto__add_index_releasefile_release_name > sentry:0454_resolve_duplicate_0452 > sentry:0455_auto__add_field_groupenvironment_first_seen > sentry:0456_auto__add_dashboard__add_unique_dashboard_organization_title__add_widg > sentry:0457_auto__add_field_savedsearch_is_global__chg_field_savedsearch_project__ > sentry:0457_auto__add_monitorcheckin__add_monitor__add_index_monitor_type_next_che > sentry:0458_global_searches_data_migration Saved Searchs: 100% |########################################################################################################################################################################| Time: 0:00:07 > sentry:0459_global_searches_unique_constraint > sentry:0460_auto__add_field_servicehook_organization_id > sentry:0461_event_attachment_indexes > sentry:0462_auto__add_servicehookproject > sentry:0462_releaseenvironment_project_id > sentry:0463_backfill_service_hook_project > sentry:0464_auto__add_sentryappcomponent__add_field_sentryapp_schema > sentry:0464_groupenvironment_foreignkeys > sentry:0465_sync > sentry:0466_auto__add_platformexternalissue__add_unique_platformexternalissue_grou > sentry:0467_backfill_integration_status > sentry:0468_auto__add_field_projectdebugfile_code_id__add_index_projectdebugfile_p > sentry:0468_recent_search > sentry:0469_fix_state > sentry:0470_org_saved_search > sentry:0471_global_saved_search_types > sentry:0472_auto__add_field_sentryapp_author The following content types are stale and need to be deleted: sentry | dsymobject sentry | dsymapp sentry | useridentity sentry | dsymsymbol sentry | dsymsdk sentry | globaldsymfile sentry | versiondsymfile sentry | projectdsymfile sentry | grouphashtombstone sentry | minidumpfile sentry | dsymbundle Any objects related to these content types by a foreign key will also be deleted. Are you sure you want to delete these content types? If you're unsure, answer 'no'. Type 'yes' to continue, or 'no' to cancel: yes Running migrations for sentry.nodestore: - Nothing to migrate. Running migrations for sentry.search: - Nothing to migrate. Running migrations for social_auth: - Nothing to migrate. Running migrations for sentry.tagstore: - Nothing to migrate. Running migrations for sentry_plugins.hipchat_ac: - Nothing to migrate. Running migrations for sentry_plugins.jira_ac: - Nothing to migrate. Synced: > django.contrib.admin > django.contrib.auth > django.contrib.contenttypes > django.contrib.messages > django.contrib.sessions > django.contrib.sites > django.contrib.staticfiles > crispy_forms > debug_toolbar > rest_framework > sentry.plugins.sentry_interface_types > sentry.plugins.sentry_mail > sentry.plugins.sentry_urls > sentry.plugins.sentry_useragents > sentry.plugins.sentry_webhooks > sudo > south Migrated: - sentry - sentry.nodestore - sentry.search - social_auth - sentry.tagstore - sentry_plugins.hipchat_ac - sentry_plugins.jira_ac Creating missing DSNs Correcting Group.num_comments counter Cleaning up... ---------------- You're all done! Run the following command get Sentry running: docker-compose up -d
Всё, версия 9.1.2 установлена.
Выводы.
Это всем известные прописные истины. Не спешите, делайте резервные копии и проверяйте всё несколько раз.
Благодарю за внимание!
ссылка на оригинал статьи https://habr.com/ru/post/512016/
Добавить комментарий