{"id":349999,"date":"2023-07-09T15:00:34","date_gmt":"2023-07-09T15:00:34","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=349999"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=349999","title":{"rendered":"<span>Dagster \u0438 Great Expectations: \u0418\u043d\u0442\u0435\u0433\u0440\u0430\u0446\u0438\u044f \u0431\u0435\u0437 \u0431\u043e\u043b\u0438<\/span>"},"content":{"rendered":"<div><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/ad6\/831\/32e\/ad683132ef7deb1cdb1806996bf04648.png\" width=\"500\" height=\"375\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/ad6\/831\/32e\/ad683132ef7deb1cdb1806996bf04648.png\"\/><\/figure>\n<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u0440\u0442\u0435\u043c \u0428\u043d\u0430\u0439\u0434\u0435\u0440, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c DataScience \u0432 \u0411\u043b\u0430\u043d\u043a\u0435. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u0432\u0430\u043c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0434\u0432\u0430 \u043c\u043e\u0449\u043d\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430 \u2013 <a href=\"https:\/\/docs.dagster.io\/getting-started?_gl=1*1ut1d7v*_ga*MTgwMjM4NTc0OC4xNjc5MzgwNDM0*_ga_84VRQZG7TV*MTY4ODU5NzExMy45LjEuMTY4ODU5NzcwOC42MC4wLjA.\" rel=\"noopener noreferrer nofollow\"><strong>Dagster<\/strong><\/a> \u0438 <a href=\"https:\/\/docs.greatexpectations.io\/docs\/\" rel=\"noopener noreferrer nofollow\"><strong>Great Expectations<\/strong><\/a>.<\/p>\n<p>Great Expectations \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u0442\u0430\u043a \u043d\u0430\u0437\u044b\u0432\u0430\u0435\u043c\u044b\u0435 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u044f \u043e\u0442 \u0432\u0430\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u0442\u043e \u0435\u0441\u0442\u044c \u0437\u0430\u0434\u0430\u0442\u044c \u043f\u0440\u0430\u0432\u0438\u043b\u0430 \u0438 \u0443\u0441\u043b\u043e\u0432\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043e\u043b\u0436\u043d\u044b \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u043e\u0432\u0430\u0442\u044c.\u00a0<\/p>\n<p>Dagster, \u0441 \u0434\u0440\u0443\u0433\u043e\u0439 \u0441\u0442\u043e\u0440\u043e\u043d\u044b, \u044d\u0442\u043e \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0441 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u043c \u043a\u043e\u0434\u043e\u043c \u0434\u043b\u044f \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u043c\u0438, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c, \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u0442\u044c \u043f\u0430\u0439\u043f\u043b\u0430\u0439\u043d\u044b \u0434\u0430\u043d\u043d\u044b\u0445.\u00a0\u041d\u0430\u043f\u0438\u0441\u0430\u043d \u043d\u0430 python, \u0447\u0442\u043e \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f\u043c \u0433\u0438\u0431\u043a\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0442\u044c \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c \u0435\u0433\u043e \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u044c. <\/p>\n<p>\u0418\u0441\u0445\u043e\u0434\u043d\u044b\u0439 \u043a\u043e\u0434 \u043a \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043d\u0430 <a href=\"https:\/\/github.com\/TemaDobryyR\/great-expectations-with-dagster.git\" rel=\"noopener noreferrer nofollow\">GitHub<\/a>.<\/p>\n<p>\u0414\u0430\u0432\u0430\u0439\u0442\u0435 \u043d\u0430\u0447\u043d\u0435\u043c?)<\/p>\n<h2>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430<\/h2>\n<p>\u041d\u0430\u043c \u043f\u043e\u043d\u0430\u0434\u043e\u0431\u044f\u0442\u0441\u044f \u043a\u0430\u043a\u0438\u0435-\u043d\u0438\u0431\u0443\u0434\u044c \u0434\u0430\u043d\u043d\u044b\u0435, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u044b \u0437\u0430\u0433\u0440\u0443\u0437\u0438\u043c \u0432 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445. \u042f \u0440\u0435\u0448\u0438\u043b \u0432\u0437\u044f\u0442\u044c <a href=\"https:\/\/www.kaggle.com\/datasets\/vipin20\/transaction-data\" rel=\"noopener noreferrer nofollow\">\u043d\u0430\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 \u0442\u0440\u0430\u043d\u0437\u0430\u043a\u0446\u0438\u0439<\/a> \u0441 Kaggle.<\/p>\n<p>\u0421\u043e\u0431\u0440\u0430\u043b \u0434\u043b\u044f \u0432\u0430\u0441 \u0433\u043e\u0442\u043e\u0432\u044b\u0439 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0439, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u043a\u043b\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043f\u043e \u044d\u0442\u043e\u0439 <a href=\"https:\/\/github.com\/TemaDobryyR\/simple-postgres-container.git\" rel=\"noopener noreferrer nofollow\"><u>\u0441\u0441\u044b\u043b\u043a\u0435<\/u><\/a>.\u00a0<\/p>\n<p>\u041f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u0432 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u044e <code>\/simple\u2011postgres\u2011container<\/code> \u0438 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u043d\u0430\u0448\u0443 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<pre><code class=\"bash\">docker compose -f \"docker-compose.yaml\" up -d --build<\/code><\/pre>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0443 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u043b\u043e\u043a\u0430\u043b\u044c\u043d\u044b\u0439 \u044d\u043a\u0437\u0435\u043c\u043f\u043b\u044f\u0440 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445 Postgres \u0441 \u0442\u0430\u0431\u043b\u0438\u0447\u043a\u043e\u0439 <em>transactions<\/em>.\u00a0<\/p>\n<p>\u0418\u0434\u0435\u043c \u0434\u0430\u043b\u044c\u0448\u0435)<\/p>\n<h2>1. Great Expectations \u0448\u0430\u0433 \u0437\u0430 \u0448\u0430\u0433\u043e\u043c<\/h2>\n<p>\u0412 \u044d\u0442\u043e\u0439 \u0447\u0430\u0441\u0442\u0438 \u043c\u044b \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u043c \u043f\u0430\u043a\u0435\u0442 <code>great-expectations<\/code>, \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u043c\u0441\u044f \u043a \u043d\u0430\u0448\u0435\u0439 \u0431\u0430\u0437\u0435 \u0434\u0430\u043d\u043d\u044b\u0445\u00a0 \u0438 \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043f\u0440\u043e\u0441\u0442\u043e\u0439 \u043d\u0430\u0431\u043e\u0440 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439.<\/p>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0443 <code>great-expectations<\/code>\u0438 \u0438\u043d\u0438\u0446\u0438\u0430\u043b\u0438\u0437\u0438\u0440\u0443\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442:<\/p>\n<pre><code class=\"bash\">pip install great-expectations great_expectations init<\/code><\/pre>\n<p>\u041a\u043e\u043c\u0430\u043d\u0434\u0430 \u0441\u043e\u0437\u0434\u0430\u0441\u0442 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u044e \u0441\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439:<\/p>\n<p><code>great_expectations\/<br \/> \u251c\u2500\u2500 checkpoints<br \/> \u251c\u2500\u2500 expectations<br \/> \u251c\u2500\u2500 great_expectations.yml<br \/> \u251c\u2500\u2500 plugins<br \/> \u2502   \u2514\u2500\u2500 custom_data_docs<br \/> \u251c\u2500\u2500 profilers<br \/> \u2514\u2500\u2500 uncommitted<br \/> \u251c\u2500\u2500 config_variables.yml<br \/> \u251c\u2500\u2500 data_docs<br \/> \u2514\u2500\u2500 validations<\/code><\/p>\n<p>\u0414\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0438\u0435 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0431\u0443\u0434\u0435\u043c \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u0438\u0437 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u0438 <code>\/great_expectations<\/code>:<\/p>\n<pre><code class=\"bash\">cd great_expectations<\/code><\/pre>\n<p>Great Expectations \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442\u00a0 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u043a\u043e\u043c\u0430\u043d\u0434\u043d\u043e\u0439 \u0441\u0442\u0440\u043e\u043a\u0438 \u0434\u043b\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u0445 \u0431\u043b\u043e\u043a\u043d\u043e\u0442\u043e\u0432 Jupyter, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u043e\u0432\u0435\u0434\u0443\u0442 \u0447\u0435\u0440\u0435\u0437 \u043a\u043e\u043d\u043a\u0440\u0435\u0442\u043d\u044b\u0435 \u044d\u0442\u0430\u043f\u044b \u0440\u0430\u0431\u043e\u0442\u044b \u0441 Great Expectations.<\/p>\n<h2>1.1 \u041d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a \u0434\u0430\u043d\u043d\u044b\u0445<\/h2>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u0443:<\/p>\n<pre><code class=\"bash\">great_expectations datasource new<\/code><\/pre>\n<p>GE \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0438\u0442 \u0432\u0430\u0440\u0438\u0430\u043d\u0442\u044b \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f \u043a \u0434\u0430\u043d\u043d\u044b\u043c, \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u043c <code>Relational database (SQL)<\/code>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/5db\/837\/233\/5db837233733277218a544e9b763c5a8.png\" width=\"547\" height=\"68\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/5db\/837\/233\/5db837233733277218a544e9b763c5a8.png\"\/><\/figure>\n<p><strong>P.S.<\/strong> \u041c\u043e\u0436\u0435\u0442 \u043f\u043e\u0442\u0440\u0435\u0431\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 &#8212; \u0438\u0445 GE \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442 \u0441\u0430\u043c \u0432 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435.<\/p>\n<p>\u0414\u0430\u043b\u0435\u0435 \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u043d\u0430\u0448\u0443 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/775\/4c3\/596\/7754c3596e77bb3d90d5184ab2a6a62a.png\" width=\"554\" height=\"186\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/775\/4c3\/596\/7754c3596e77bb3d90d5184ab2a6a62a.png\"\/><\/figure>\n<p>GE \u0441\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u0442 Jupyter \u043d\u043e\u0443\u0442\u0431\u0443\u043a, \u0432 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u043c\u044b \u0434\u043e\u043b\u0436\u043d\u044b \u0437\u0430\u043f\u043e\u043b\u043d\u0438\u0442\u044c \u0443\u0447\u0435\u0442\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043b\u044f \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/8cd\/6bc\/3e4\/8cd6bc3e48d3f88590d7c23b825a66ef.png\" width=\"1084\" height=\"625\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/8cd\/6bc\/3e4\/8cd6bc3e48d3f88590d7c23b825a66ef.png\"\/><\/figure>\n<p>\u0411\u043e\u043b\u044c\u0448\u0435 \u043d\u0438\u0447\u0435\u0433\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043d\u0435 \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f, \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u043c \u043d\u043e\u0443\u0442\u0431\u0443\u043a \u0434\u043e \u043a\u043e\u043d\u0446\u0430, \u0437\u0430\u043a\u0440\u044b\u0432\u0430\u0435\u043c \u0438 \u043f\u0440\u0435\u0440\u044b\u0432\u0430\u0435\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u0443 \u0432 \u0442\u0435\u0440\u043c\u0438\u043d\u0430\u043b\u0435. \u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e \u043d\u0430\u0448\u0435\u0433\u043e \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u0430 \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u043e\u0436\u043d\u043e \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0432 \u0444\u0430\u0439\u043b\u0435 <code>great_expectations.yaml.<\/code><\/p>\n<details class=\"spoiler\">\n<summary>great_expectations.yaml<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"yaml\"># Welcome to Great Expectations! Always know what to expect from your data. # # Here you can define datasources, batch kwargs generators, integrations and # more. This file is intended to be committed to your repo. For help with # configuration please: #   - Read our docs: https:\/\/docs.greatexpectations.io\/docs\/guides\/connecting_to_your_data\/connect_to_data_overview\/#2-configure-your-datasource #   - Join our slack channel: http:\/\/greatexpectations.io\/slack  # config_version refers to the syntactic version of this config file, and is used in maintaining backwards compatibility # It is auto-generated and usually does not need to be changed. config_version: 3.0  # Datasources tell Great Expectations where your data lives and how to get it. # You can use the CLI command `great_expectations datasource new` to help you # add a new datasource. Read more at https:\/\/docs.greatexpectations.io\/docs\/guides\/connecting_to_your_data\/connect_to_data_overview datasources:   my_datasource:     class_name: Datasource     module_name: great_expectations.datasource     data_connectors:       default_runtime_data_connector_name:         batch_identifiers:           - default_identifier_name         class_name: RuntimeDataConnector         module_name: great_expectations.datasource.data_connector       default_inferred_data_connector_name:         include_schema_name: true         class_name: InferredAssetSqlDataConnector         module_name: great_expectations.datasource.data_connector         introspection_directives:           schema_name: public       default_configured_data_connector_name:         class_name: ConfiguredAssetSqlDataConnector         assets:           transactions:             class_name: Asset             schema_name: public             module_name: great_expectations.datasource.data_connector.asset         module_name: great_expectations.datasource.data_connector     execution_engine:       class_name: SqlAlchemyExecutionEngine       credentials:         host: localhost         port: '5432'         username: postgres         password: postgres         database: postgres         drivername: postgresql       module_name: great_expectations.execution_engine  # This config file supports variable substitution which enables: 1) keeping # secrets out of source control &amp; 2) environment-based configuration changes # such as staging vs prod. # # When GX encounters substitution syntax (like `my_key: ${my_value}` or # `my_key: $my_value`) in the great_expectations.yml file, it will attempt # to replace the value of `my_key` with the value from an environment # variable `my_value` or a corresponding key read from this config file, # which is defined through the `config_variables_file_path`. # Environment variables take precedence over variables defined here. # # Substitution values defined here can be a simple (non-nested) value, # nested value such as a dictionary, or an environment variable (i.e. ${ENV_VAR}) # # # https:\/\/docs.greatexpectations.io\/docs\/guides\/setup\/configuring_data_contexts\/how_to_configure_credentials   config_variables_file_path: uncommitted\/config_variables.yml  # The plugins_directory will be added to your python path for custom modules # used to override and extend Great Expectations. plugins_directory: plugins\/  stores: # Stores are configurable places to store things like Expectations, Validations # Data Docs, and more. These are for advanced users only - most users can simply # leave this section alone. # # Three stores are required: expectations, validations, and # evaluation_parameters, and must exist with a valid store entry. Additional # stores can be configured for uses such as data_docs, etc.   expectations_store:     class_name: ExpectationsStore     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: expectations\/    validations_store:     class_name: ValidationsStore     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: uncommitted\/validations\/    evaluation_parameter_store:     class_name: EvaluationParameterStore   checkpoint_store:     class_name: CheckpointStore     store_backend:       class_name: TupleFilesystemStoreBackend       suppress_store_backend_id: true       base_directory: checkpoints\/    profiler_store:     class_name: ProfilerStore     store_backend:       class_name: TupleFilesystemStoreBackend       suppress_store_backend_id: true       base_directory: profilers\/  expectations_store_name: expectations_store validations_store_name: validations_store evaluation_parameter_store_name: evaluation_parameter_store checkpoint_store_name: checkpoint_store  data_docs_sites:   # Data Docs make it simple to visualize data quality in your project. These   # include Expectations, Validations &amp; Profiles. The are built for all   # Datasources from JSON artifacts in the local repo including validations &amp;   # profiles from the uncommitted directory. Read more at https:\/\/docs.greatexpectations.io\/docs\/terms\/data_docs   local_site:     class_name: SiteBuilder     show_how_to_buttons: true     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: uncommitted\/data_docs\/local_site\/     site_index_builder:       class_name: DefaultSiteIndexBuilder  anonymous_usage_statistics:   enabled: true   data_context_id: ffd790a7-4454-4b53-b9b1-428bfcfb4e64 notebooks: include_rendered_content:   globally: false   expectation_validation_result: false   expectation_suite: false <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>1.2 \u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043d\u0430\u0431\u043e\u0440 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439<\/h2>\n<p>\u0412\u043e\u0437\u0432\u0440\u0430\u0449\u0430\u0435\u043c\u0441\u044f \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0443\u044e \u0441\u0442\u0440\u043e\u043a\u0443 \u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u043c:<\/p>\n<pre><code class=\"bash\">great_expectations suite new<\/code><\/pre>\n<p>GE \u043d\u0430 \u0432\u044b\u0431\u043e\u0440 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u0442 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0441\u043f\u043e\u0441\u043e\u0431\u043e\u0432 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0431\u043e\u0440\u043e\u0432 \u043f\u0440\u043e\u0432\u0435\u0440\u043e\u043a. \u0412 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438 \u0440\u0435\u043a\u043e\u043c\u0435\u043d\u0434\u0443\u0435\u0442\u0441\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0441 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435\u043c <a href=\"https:\/\/docs.greatexpectations.io\/docs\/terms\/data_assistant\/\" rel=\"noopener noreferrer nofollow\">Data Assistant<\/a> \u2014 \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u043e\u0439 \u0443\u0442\u0438\u043b\u0438\u0442\u044b, \u0443\u043f\u0440\u043e\u0449\u0430\u044e\u0449\u0435\u0439 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439.<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/51d\/0a4\/6d3\/51d0a46d345426b62d98ba19fc683e16.png\" width=\"619\" height=\"98\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/51d\/0a4\/6d3\/51d0a46d345426b62d98ba19fc683e16.png\"\/><\/figure>\n<p>\u0422\u0430\u043a\u0436\u0435 GE \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442 \u0442\u0440\u0438 \u0442\u0438\u043f\u0430 <code>DataConnector<\/code> \u043a\u043b\u0430\u0441\u0441\u043e\u0432. \u041f\u0440\u043e \u0432\u044b\u0431\u043e\u0440 \u043a\u043e\u043d\u043d\u0435\u043a\u0442\u043e\u0440\u043e\u0432 \u043c\u043e\u0436\u043d\u043e \u043f\u043e\u0447\u0438\u0442\u0430\u0442\u044c <a href=\"https:\/\/docs.greatexpectations.io\/docs\/0.15.50\/guides\/connecting_to_your_data\/how_to_choose_which_dataconnector_to_use\/\" rel=\"noopener noreferrer nofollow\"><u>\u0437\u0434\u0435\u0441\u044c<\/u><\/a>, \u0430 \u043d\u0430\u043c \u043d\u0443\u0436\u0435\u043d <code>default_configured_data_connector_name<\/code>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/a6b\/278\/858\/a6b2788586684e6f792f03932019b6d3.png\" width=\"646\" height=\"131\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/a6b\/278\/858\/a6b2788586684e6f792f03932019b6d3.png\"\/><\/figure>\n<p>\u0412\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u0430\u0441\u0441\u0435\u0442 \u0434\u0430\u043d\u043d\u044b\u0445 \u0438 \u0437\u0430\u0434\u0430\u0435\u043c \u0438\u043c\u044f \u043d\u0430\u0448\u0435\u043c\u0443 \u043d\u0430\u0431\u043e\u0440\u0443 \u043f\u0440\u043e\u0432\u0435\u0440\u043e\u043a:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/506\/7ba\/9a6\/5067ba9a6194402fe8070c9e61923c31.png\" width=\"790\" height=\"163\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/506\/7ba\/9a6\/5067ba9a6194402fe8070c9e61923c31.png\"\/><\/figure>\n<p>\u041d\u0430\u0431\u043e\u0440\u044b \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439, \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0435 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043d\u043e\u0443\u0442\u0431\u0443\u043a\u0430, \u043d\u0435 \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u044b \u0434\u043b\u044f \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0439 \u0441\u0440\u0435\u0434\u0435. \u041c\u044b \u043c\u043e\u0436\u0435\u043c \u043e\u0442\u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0441\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b<code>\/expectations\/transactions\/suite.json<\/code> \u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0442\u0443\u0434\u0430 \u0435\u0449\u0435 \u043f\u0430\u0440\u0443 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439: \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u043c, \u0447\u0442\u043e \u0432\u0441\u0435 \u0442\u0440\u0430\u043d\u0437\u0430\u043a\u0446\u0438\u0438 \u0438\u043c\u0435\u044e\u0442 ID, \u0430 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u043e \u043a\u0443\u043f\u043b\u0435\u043d\u043d\u044b\u0445 \u0442\u043e\u0432\u0430\u0440\u043e\u0432 \u0431\u043e\u043b\u044c\u0448\u0435 0. <\/p>\n<details class=\"spoiler\">\n<summary>\u0418\u0442\u043e\u0433\u043e\u0432\u044b\u0439 suite.json<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"yaml\">{   \"data_asset_type\": null,   \"expectation_suite_name\": \"transactions.suite\",   \"expectations\": [     {       \"expectation_type\": \"expect_table_row_count_to_be_between\",       \"kwargs\": {         \"max_value\": 2000000,         \"min_value\": 0       },       \"meta\": {         \"profiler_details\": {           \"metric_configuration\": {             \"domain_kwargs\": {},             \"metric_name\": \"table.row_count\",             \"metric_value_kwargs\": null           },           \"num_batches\": 1         }       }     },     {       \"expectation_type\": \"expect_table_columns_to_match_set\",       \"kwargs\": {         \"column_set\": [           \"ItemCode\",           \"UserId\",           \"NumberOfItemsPurchased\",           \"Country\",           \"TransactionTime\",           \"ItemDescription\",           \"TransactionId\",           \"CostPerItem\"         ],         \"exact_match\": null       },       \"meta\": {         \"profiler_details\": {           \"success_ratio\": 1.0         }       }     },     {       \"expectation_type\": \"expect_column_values_to_not_be_null\",       \"kwargs\": {         \"column\": \"TransactionId\"       }     },     {       \"expectation_type\": \"expect_column_values_to_be_between\",       \"kwargs\": {         \"column\": \"NumberOfItemsPurchased\",         \"min_value\": 1       }     }   ],   \"ge_cloud_id\": null,   \"meta\": {     \"citations\": [       {         \"citation_date\": \"2023-07-06T00:08:01.886955Z\",         \"comment\": \"Created by effective Rule-Based Profiler of OnboardingDataAssistant with the configuration included.\\n\"       }     ],     \"great_expectations_version\": \"0.17.2\"   } }<\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0421 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e\u043c \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u044b\u0445 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439 \u043c\u043e\u0436\u043d\u043e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0432 <a href=\"https:\/\/greatexpectations.io\/expectations\/?filterType=Backend+support&amp;viewType=Summary&amp;showFilters=true&amp;subFilterValues=postgresql\" rel=\"noopener noreferrer nofollow\"><u>Expectations Store<\/u><\/a><u>.<\/u><\/p>\n<p>\u0422\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u0443 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u0434\u0432\u0435 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u043d\u0430 \u0443\u0440\u043e\u0432\u043d\u0435 \u0442\u0430\u0431\u043b\u0438\u0446\u044b, \u0438 \u0435\u0449\u0435 \u0434\u0432\u0435 \u2014 \u043d\u0430 \u0443\u0440\u043e\u0432\u043d\u0435 \u043a\u043e\u043b\u043e\u043d\u043e\u043a. \u0418\u0434\u0435\u043c \u0434\u0430\u043b\u044c\u0448\u0435.<\/p>\n<h2>1.3 \u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 Checkpoint<\/h2>\n<p><strong>Checkpoint (\u043a\u043e\u043d\u0442\u0440\u043e\u043b\u044c\u043d\u0430\u044f \u0442\u043e\u0447\u043a\u0430) <\/strong>\u2014 \u044d\u0442\u043e \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432\u043e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 \u043f\u0440\u0438 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0438 Great Expectations \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0439 \u0441\u0440\u0435\u0434\u0435. \u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u0443:<\/p>\n<pre><code class=\"bash\">great_expectations checkpoint new my_checkpoint<\/code><\/pre>\n<p>\u041f\u043e\u0441\u043b\u0435 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0432\u0441\u0435\u0445 \u0448\u0430\u0433\u043e\u0432 \u0432 \u043d\u043e\u0443\u0442\u0431\u0443\u043a\u0435 \u0432 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u0438 <code>\/checkpoints<\/code> \u043f\u043e\u044f\u0432\u0438\u0442\u0441\u044f \u0444\u0430\u0439\u043b <code>my_checkpoint.yaml:<\/code><\/p>\n<details class=\"spoiler\">\n<summary>my_checkpoint.yaml<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"yaml\">name: my_checkpoint config_version: 1.0 template_name: module_name: great_expectations.checkpoint class_name: Checkpoint run_name_template: '%Y%m%d-%H%M%S-my-run-name-template' expectation_suite_name: batch_request: {} action_list:   - name: store_validation_result     action:       class_name: StoreValidationResultAction   - name: store_evaluation_params     action:       class_name: StoreEvaluationParametersAction   - name: update_data_docs     action:       class_name: UpdateDataDocsAction       site_names: [] evaluation_parameters: {} runtime_configuration: {} validations:   - batch_request:       datasource_name: my_datasource       data_connector_name: default_inferred_data_connector_name       data_asset_name: public.transactions       data_connector_query:         index: -1     expectation_suite_name: transactions.suite profilers: [] ge_cloud_id: expectation_suite_ge_cloud_id: <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<p>\u0415\u0441\u043b\u0438 \u0432 \u043d\u043e\u0443\u0442\u0431\u0443\u043a\u0435 \u0432\u044b \u043d\u0435 \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u043b\u0438 \u0440\u0430\u0431\u043e\u0442\u043e\u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c \u0447\u0435\u043a\u043f\u043e\u0438\u043d\u0442\u0430, \u044d\u0442\u043e \u043c\u043e\u0436\u043d\u043e \u0441\u0434\u0435\u043b\u0430\u0442\u044c \u0432 \u0442\u0435\u0440\u043c\u0438\u043d\u0430\u043b\u0435:<\/p>\n<pre><code class=\"bash\">great_expectations checkpoint run my_checkpoint<\/code><\/pre>\n<p>\u041c\u044b \u0434\u043e\u043b\u0436\u043d\u044b \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u044b \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/bd7\/b25\/366\/bd7b253669600e332d37cd3dca537376.png\" width=\"599\" height=\"65\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/bd7\/b25\/366\/bd7b253669600e332d37cd3dca537376.png\"\/><\/figure>\n<p>\u0412 \u0442\u0435\u0440\u043c\u0438\u043d\u0430\u043b\u0435 \u043d\u0435 \u0431\u0443\u0434\u0435\u0442 \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0430\u0442\u044c\u0441\u044f \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044f \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0430\u044f \u0438\u043c\u0435\u043d\u043d\u043e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0430 \u043d\u0435 \u043f\u0440\u043e\u0448\u043b\u0430, \u043e\u0434\u043d\u0430\u043a\u043e \u0438\u043c\u0435\u0435\u0442\u0441\u044f \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u0441\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044e \u0432 \u0432\u0438\u0434\u0435 \u0444\u0430\u0439\u043b\u043e\u0432 HTML, \u0447\u0442\u043e \u043f\u043e\u0437\u0432\u043e\u043b\u0438\u0442 \u043f\u043e\u0441\u043c\u043e\u0442\u0440\u0435\u0442\u044c \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u044b \u0432 \u043b\u044e\u0431\u043e\u043c \u0431\u0440\u0430\u0443\u0437\u0435\u0440\u0435. \u041f\u043e\u0434\u0440\u043e\u0431\u043d\u0435\u0435 \u043e\u0431 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0438 <strong>Data Docs<\/strong> \u043c\u043e\u0436\u043d\u043e \u043f\u043e\u0447\u0438\u0442\u0430\u0442\u044c <a href=\"https:\/\/docs.greatexpectations.io\/docs\/terms\/data_docs\/\" rel=\"noopener noreferrer nofollow\"><u>\u0437\u0434\u0435\u0441\u044c<\/u><\/a>.<\/p>\n<pre><code class=\"bash\">great_expectations docs build<\/code><\/pre>\n<p>\u041a\u043e\u043c\u0430\u043d\u0434\u0430 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u043e\u0442\u043a\u0440\u043e\u0435\u0442 \u043e\u043a\u043d\u043e \u0432 \u0431\u0440\u0430\u0443\u0437\u0435\u0440\u0435, \u0447\u0442\u043e\u0431\u044b \u043c\u044b \u0441\u043c\u043e\u0433\u043b\u0438 \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0440\u0435\u0437\u0443\u043b\u0442\u0430\u0442\u044b \u0432\u0430\u043b\u0438\u0434\u0430\u0446\u0438\u0438:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/e5a\/d98\/379\/e5ad983797350a1d63ef249d6f232f86.png\" width=\"1600\" height=\"398\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/e5a\/d98\/379\/e5ad983797350a1d63ef249d6f232f86.png\"\/><\/figure>\n<p>\u041d\u0430\u0431\u043e\u0440 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439 \u0433\u043e\u0442\u043e\u0432, \u0430 \u0437\u043d\u0430\u0447\u0438\u0442, \u043c\u044b \u0433\u043e\u0442\u043e\u0432\u044b \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0442\u044c \u043d\u0430\u0448\u0438 \u0447\u0435\u043a\u043f\u043e\u0438\u043d\u0442\u044b \u0432 Dagster!<\/p>\n<h2>2. \u0420\u0430\u0431\u043e\u0442\u0430 \u0441 Dagster<\/h2>\n<p>\u041f\u0440\u0435\u0436\u0434\u0435, \u0447\u0435\u043c \u043c\u044b \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043d\u0430\u0448 \u043f\u0440\u043e\u0435\u043a\u0442, \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043c \u043e\u0431 \u043e\u0441\u043d\u043e\u0432\u043d\u044b\u0445 \u043a\u043e\u043d\u0446\u0435\u043f\u0446\u0438\u044f\u0445 Dagster.<\/p>\n<p><strong>Asset (\u0430\u043a\u0442\u0438\u0432\u044b)<\/strong> \u2014 \u044d\u0442\u043e \u043e\u0431\u044a\u0435\u043a\u0442, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0444\u0438\u043a\u0441\u0438\u0440\u0443\u0435\u0442 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u043d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0433\u043e \u044d\u0442\u0430\u043f\u0430 \u0432 \u043f\u0430\u0439\u043f\u043b\u0430\u0439\u043d\u0435 dagster. \u0410\u0441\u0441\u0435\u0442\u044b \u043c\u043e\u0433\u0443\u0442 \u043b\u044e\u0431\u043e\u0433\u043e \u0442\u0438\u043f\u0430, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0444\u0430\u0439\u043b\u0430\u043c\u0438, \u0442\u0430\u0431\u043b\u0438\u0446\u0430\u043c\u0438 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u043b\u0438 \u043c\u043e\u0434\u0435\u043b\u044c\u044e \u043c\u0430\u0448\u0438\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f.<\/p>\n<p><strong>Ops (\u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0438)<\/strong> \u2014 \u044f\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u0435\u0434\u0438\u043d\u0438\u0446\u0435\u0439 \u0432\u044b\u0447\u0438\u0441\u043b\u0435\u043d\u0438\u044f \u0432 Dagster, \u0438 \u0434\u043e\u043b\u0436\u043d\u044b \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u043e\u0442\u043d\u043e\u0441\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043f\u0440\u043e\u0441\u0442\u044b\u0435 \u0437\u0430\u0434\u0430\u0447\u0438, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u043a\u0430\u043a\u0438\u0435-\u043d\u0438\u0431\u0443\u0434\u044c \u0432\u044b\u0447\u0438\u0441\u043b\u0435\u043d\u0438\u044f, \u043e\u0431\u0440\u0430\u0449\u0435\u043d\u0438\u044f \u043a \u0431\u0430\u0437\u0435 \u0434\u0430\u043d\u043d\u044b\u0445, \u0432\u044b\u0437\u043e\u0432\u044b API \u0438 \u0442.\u0434.<\/p>\n<p>\u0422\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0438 \u2014 \u044d\u0442\u043e \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u044e\u0442\u0441\u044f \u0432 \u043f\u0430\u0439\u043f\u043b\u0430\u0439\u043d\u0435, \u0430 \u0430\u043a\u0442\u0438\u0432\u044b \u2014 \u044d\u0442\u043e \u0434\u0430\u043d\u043d\u044b\u0435, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u044e\u0442\u0441\u044f \u0432 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u044d\u0442\u0438\u0445 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0439.\u00a0<\/p>\n<p><strong>Jobs (\u0437\u0430\u0434\u0430\u043d\u0438\u044f)<\/strong> &#8212; \u043e\u0441\u043d\u043e\u0432\u043d\u0430\u044f \u0435\u0434\u0438\u043d\u0438\u0446\u0430 \u0438\u0441\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430 \u0432 Dagster. \u0417\u0430\u0434\u0430\u043d\u0438\u0435 \u043c\u043e\u0436\u0435\u0442 \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u0442\u044c \u043d\u0430\u0431\u043e\u0440 \u0430\u043a\u0442\u0438\u0432\u043e\u0432, \u0438\u043b\u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0442\u044c \u0433\u0440\u0430\u0444 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0439.<\/p>\n<p><strong>Resources (\u0440\u0435\u0441\u0443\u0440\u0441\u044b)<\/strong> &#8212; \u044d\u0442\u043e \u043e\u0431\u044a\u0435\u043a\u0442\u044b, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u044f\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043e\u0431\u0449\u0438\u043c\u0438 \u0434\u043b\u044f \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u0430\u043a\u0442\u0438\u0432\u043e\u0432, \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0439, \u0440\u0430\u0441\u043f\u0438\u0441\u0430\u043d\u0438\u0439 \u0438\u043b\u0438 \u0434\u0430\u0442\u0447\u0438\u043a\u043e\u0432. \u041d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u043c \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 \u043a \u0431\u0430\u0437\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u043b\u0438 \u0441\u0435\u0440\u0432\u0438\u0441\u0443.<\/p>\n<h2>2.1 \u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u043f\u0440\u043e\u0435\u043a\u0442\u0430<\/h2>\n<p>\u0421\u0430\u043c\u044b\u0439 \u043f\u0440\u043e\u0441\u0442\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431 \u043d\u0430\u0447\u0430\u0442\u044c \u0440\u0430\u0431\u043e\u0442\u0443 \u0441 Dagster \u2014 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0448\u0430\u0431\u043b\u043e\u043d \u043f\u0440\u043e\u0435\u043a\u0442\u0430 \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u0430 \u043a\u043e\u043c\u0430\u043d\u0434\u043d\u043e\u0439 \u0441\u0442\u0440\u043e\u043a\u0438:<\/p>\n<pre><code class=\"bash\">pip install dagster dagster project scaffold --name great-expectations-dagster<\/code><\/pre>\n<p>\u041a\u043e\u043c\u0430\u043d\u0434\u0430 <code>dagster project<\/code> \u0441\u043e\u0437\u0434\u0430\u0441\u0442 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u0443\u044e \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0443 \u043f\u0430\u043f\u043e\u043a:<\/p>\n<p><code>great-expectations-dagster\/<br \/> \u251c\u2500\u2500 great_expectations_dagster<br \/> \u2502   \u251c\u2500\u2500 assets.py<br \/> \u2502   \u2514\u2500\u2500 init.py<br \/> \u251c\u2500\u2500 great_expectations_dagster_tests\/<br \/> \u251c\u2500\u2500 pyproject.toml<br \/> \u251c\u2500\u2500 README.md<br \/> \u251c\u2500\u2500 setup.cfg<br \/> \u2514\u2500\u2500 setup.py<\/code><\/p>\n<p>\u0424\u0430\u0439\u043b\u044b, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c\u0438 \u043c\u044b \u0431\u0443\u0434\u0435\u043c \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c:<\/p>\n<ul>\n<li>\n<p><code>__init__.py<\/code> \u2014 \u0441\u043e\u0434\u0435\u0440\u0436\u0438\u0442 \u0432\u0441\u0435 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f, \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0435 \u0432 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442\u0435. \u042d\u0442\u043e \u043c\u043e\u0433\u0443\u0442 \u0430\u043a\u0442\u0438\u0432\u044b, \u0437\u0430\u0434\u0430\u043d\u0438\u044f, \u0440\u0430\u0441\u043f\u0438\u0441\u0430\u043d\u0438\u044f, \u0434\u0430\u0442\u0447\u0438\u043a\u0438 \u0438 \u0440\u0435\u0441\u0443\u0440\u0441\u044b;<\/p>\n<\/li>\n<li>\n<p><code>assets.py<\/code> \u2014 \u043c\u043e\u0434\u0443\u043b\u044c Python, \u0441\u043e\u0434\u0435\u0440\u0436\u0430\u0449\u0438\u0439 \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u043d\u043e-\u043e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c\u044b\u0435 \u0430\u043a\u0442\u0438\u0432\u044b;<\/p>\n<\/li>\n<li>\n<p><code>setup.py<\/code> \u2014 \u0441\u043e\u0434\u0435\u0440\u0436\u0438\u0442 \u0441\u0446\u0435\u043d\u0430\u0440\u0438\u0439 \u0441\u0431\u043e\u0440\u043a\u0438, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u0435\u0433\u043e \u0434\u043b\u044f \u0443\u043a\u0430\u0437\u0430\u043d\u0438\u044f \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439 \u0434\u043b\u044f \u043d\u0430\u0448\u0435\u0433\u043e \u043f\u0440\u043e\u0435\u043a\u0442\u0430.<\/p>\n<\/li>\n<\/ul>\n<p>\u041f\u043e\u0434\u0440\u043e\u0431\u043d\u0435\u0435 \u043e \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0435 \u043f\u0440\u043e\u0435\u043a\u0442\u0430 Dagster \u043c\u043e\u0436\u043d\u043e \u043f\u0440\u043e\u0447\u0438\u0442\u0430\u0442\u044c <a href=\"https:\/\/docs.dagster.io\/getting-started\/project-file-reference\" rel=\"noopener noreferrer nofollow\"><u>\u0437\u0434\u0435\u0441\u044c<\/u><\/a>.<\/p>\n<h2>2.2 \u0421\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u043d\u043e-\u043e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0435\u043c\u043e\u0433\u043e \u0430\u043a\u0442\u0438\u0432\u0430<\/h2>\n<p>\u041c\u044b \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0430\u0441\u0441\u0435\u0442, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u0440\u043e\u0432\u0430\u043b\u0438\u0434\u0438\u043b\u0438\u0440\u0443\u0435\u0442 \u043d\u0430\u0448\u0438 \u0434\u0430\u043d\u043d\u044b\u0435 \u0438 \u0437\u0430\u043f\u0438\u0448\u0435\u0442 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u044b \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0432 \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435.<\/p>\n<p><strong>P.S. <\/strong>\u0412 Dagster \u0435\u0441\u0442\u044c \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0444\u0430\u0431\u0440\u0438\u043a\u0443 <code>ge_validation_op_factory<\/code> \u0438\u0437 \u043f\u0430\u043a\u0435\u0442\u0430 <a href=\"https:\/\/docs.dagster.io\/integrations\/great-expectations\" rel=\"noopener noreferrer nofollow\">dagster-ge<\/a> \u0434\u043b\u044f \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0439 Dagster, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u0443\u044e\u0442\u0441\u044f \u0441 Great Expectations. \u041e\u0434\u043d\u0430\u043a\u043e \u044d\u0442\u043e \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0438, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0432\u044b\u0433\u0440\u0443\u0437\u0438\u0442 \u043d\u0430\u043c \u043d\u0430\u0448 \u043d\u0430\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 \u0432 \u0432\u0438\u0434\u0435 \u0434\u0430\u0442\u0430\u0444\u0440\u0435\u0439\u043c\u0430 \u0438 \u043f\u0435\u0440\u0435\u0434\u0430\u0441\u0442 \u0435\u0433\u043e \u0432 \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u0435 \u0432\u0445\u043e\u0434\u043d\u044b\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u0438, \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u043e\u0439 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e <code>ge_validation_op_factory.<\/code><\/p>\n<p>\u041e\u0442\u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u0443\u0435\u043c \u0444\u0430\u0439\u043b <code>assets.py:<\/code><\/p>\n<pre><code class=\"python\">from dagster import MetadataValue, Output, asset from dagster_ge.factory import GEContextResource from great_expectations.render.renderer import ValidationResultsPageRenderer from great_expectations.render.view import DefaultMarkdownPageView   @asset def validate_data(data_context: GEContextResource):      # \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u043c \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442 \u0434\u0430\u043d\u043d\u044b\u0445 GE     context = data_context.get_data_context()      # \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0443 \u043d\u0430 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0438\u0435 \u043d\u0430\u0431\u043e\u0440\u0443 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439     results = context.run_checkpoint('my_checkpoint')      # \u0432\u0438\u0437\u0443\u0430\u0437\u043b\u0438\u0446\u0438\u044f \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u043e\u0432 \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0432 Dagster \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0432\u044b\u0432\u043e\u0434\u0430 \u043c\u0435\u0442\u0430\u0434\u0430\u043d\u043d\u044b\u0445       validation_results_page_renderer = ValidationResultsPageRenderer(run_info_at_end=True)      rendered_document_content_list = (         validation_results_page_renderer.render_validation_operator_result(results)     )     md_str = \" \".join(DefaultMarkdownPageView().render(rendered_document_content_list))      return Output(         value=results['success'],         metadata={             \"Expectation Results\": MetadataValue.md(md_str)     })<\/code><\/pre>\n<p>\u0414\u0430\u043b\u0435\u0435 \u0438\u0434\u0435\u043c \u0432 <code>__init__.py<\/code> \u0438 \u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u0443\u0435\u043c \u043d\u0430\u0448\u0435 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u0435:<\/p>\n<pre><code class=\"python\">from dagster import (Definitions, define_asset_job, file_relative_path,                      load_assets_from_modules) from dagster_ge.factory import GEContextResource from great_expectations_dagster.assets import validate_data    defs = Definitions(     assets=[         validate_data     ],     jobs=[         # \u0437\u0430\u0434\u0430\u043d\u0438\u0435, \u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442 \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u044e \u043d\u0430\u0448\u0435\u0433\u043e \u0430\u0441\u0441\u0435\u0442\u0430         define_asset_job(             name='validate_data_job',             selection=[validate_data]         )     ],     resources={         # \u0434\u043b\u044f \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u0430 GE \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u043a\u043e\u0440\u0435\u043d\u044c \u043a\u0430\u0442\u0430\u043b\u043e\u0433\u0430 GE         # (\u043f\u0443\u0442\u044c \u043a \u0444\u0430\u0439\u043b\u0443 great_exepctations.yaml)         'data_context': GEContextResource(             ge_root_dir=file_relative_path(__file__, '..\/..\/great_expectations')         )     } )<\/code><\/pre>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043d\u0430\u043c \u043e\u0441\u0442\u0430\u043d\u0435\u0442\u0441\u044f \u043f\u043e\u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0444\u0430\u0439\u043b <code>setup.py<\/code> \u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0442\u0443\u0434\u0430 \u043f\u0430\u043a\u0435\u0442 <code>dagster-ge<\/code>:<\/p>\n<pre><code class=\"python\">from setuptools import find_packages, setup  setup(     name=\"great_expectations_dagster\",     packages=find_packages(exclude=[\"great_expectations_dagster_tests\"]),     install_requires=[         \"dagster\",         \"dagster-ge\"     ],     extras_require={\"dev\": [\"dagit\", \"pytest\"]}, ) <\/code><\/pre>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0432\u0441\u0435 \u0433\u043e\u0442\u043e\u0432\u043e, \u0447\u0442\u043e\u0431\u044b \u043c\u044b \u043c\u043e\u0433\u043b\u0438 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c \u043d\u0430\u0448 \u043c\u0430\u043b\u0435\u043d\u044c\u043a\u0438\u0439 \u043f\u0430\u043f\u043b\u0430\u0439\u043d!<\/p>\n<p>\u0412 \u043a\u043e\u0440\u043d\u0435 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u0438 <code>great-expectations-dagster\/<\/code> \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0435 \u043a\u043e\u043c\u0430\u043d\u0434\u044b:<\/p>\n<pre><code class=\"bash\">pip install -e \u201c.[dev]\u201d dagit<\/code><\/pre>\n<p>\u041a\u043e\u043c\u0430\u043d\u0434\u0430 <code>dagit<\/code> \u0440\u0430\u0441\u043f\u0435\u0447\u0430\u0442\u0430\u0435\u0442 URL-\u0430\u0434\u0440\u0435\u0441, \u043f\u043e \u043a\u043e\u0442\u043e\u0440\u043e\u043c\u0443 \u043c\u044b \u0441\u043c\u043e\u0436\u0435\u043c \u043f\u043e\u043b\u0443\u0447\u0438\u0442\u044c \u0434\u043e\u0441\u0442\u0443\u043f \u043a \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441\u0443 \u0432 \u0431\u0440\u0430\u0443\u0437\u0435\u0440\u0435, \u043e\u0431\u044b\u0447\u043d\u043e \u0447\u0435\u0440\u0435\u0437 \u043f\u043e\u0440\u0442 3000. \u041c\u044b \u0434\u043e\u043b\u0436\u043d\u044b \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u043d\u0430\u0448\u0435 \u0437\u0430\u0434\u0430\u043d\u0438\u0435 <code>validate_data_job<\/code>:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/1d3\/fdb\/e9f\/1d3fdbe9fbd19f9e615aae893270fefe.png\" width=\"1600\" height=\"900\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/1d3\/fdb\/e9f\/1d3fdbe9fbd19f9e615aae893270fefe.png\"\/><\/figure>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043d\u0430\u043c \u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u043e \u043d\u0430\u0436\u0430\u0442\u044c \u043d\u0430 \u043a\u043d\u043e\u043f\u043a\u0443 <strong>Materialize<\/strong>, \u043f\u043e\u0434\u043e\u0436\u0434\u0430\u0442\u044c, \u043f\u043e\u043a\u0430 \u0437\u0430\u0434\u0430\u043d\u0438\u0435 \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0438\u0437\u0443\u0435\u0442 \u0430\u043a\u0442\u0438\u0432, \u0438 \u043c\u044b \u0441\u043c\u043e\u0436\u0435\u043c \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u043c\u0435\u0442\u0430\u0434\u0430\u043d\u043d\u044b\u0435 \u043d\u0430\u0448\u0435\u0439 \u043c\u0430\u0442\u0435\u0440\u0438\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438:<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/f74\/b18\/8eb\/f74b188eb8be62f89f10bcc78b901d58.png\" alt=\"\u0414\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u043f\u0430\u043d\u0435\u043b\u044c \u0441\u043f\u0440\u0430\u0432\u0430, \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043a\u043b\u0438\u043a\u043d\u0443\u0442\u044c \u043d\u0430 \u0438\u043a\u043e\u043d\u0443 \u0430\u0441\u0441\u0435\u0442\u0430\" title=\"\u0414\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u043f\u0430\u043d\u0435\u043b\u044c \u0441\u043f\u0440\u0430\u0432\u0430, \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043a\u043b\u0438\u043a\u043d\u0443\u0442\u044c \u043d\u0430 \u0438\u043a\u043e\u043d\u0443 \u0430\u0441\u0441\u0435\u0442\u0430\" width=\"1600\" height=\"900\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/f74\/b18\/8eb\/f74b188eb8be62f89f10bcc78b901d58.png\"\/><\/p>\n<div><figcaption>\u0414\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u043f\u0430\u043d\u0435\u043b\u044c \u0441\u043f\u0440\u0430\u0432\u0430, \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043a\u043b\u0438\u043a\u043d\u0443\u0442\u044c \u043d\u0430 \u0438\u043a\u043e\u043d\u0443 \u0430\u0441\u0441\u0435\u0442\u0430<\/figcaption><\/div>\n<\/figure>\n<p>\u041d\u0430 \u0432\u043a\u043b\u0430\u0434\u043a\u0435 \u043c\u044b \u0432\u0438\u0434\u0438\u043c \u043f\u043e\u043b\u0435 <strong>Expectation Results<\/strong> \u0438 \u0441\u0441\u044b\u043b\u043a\u0443 <strong><u>[Show Markdown]<\/u><\/strong>,<strong> <\/strong>\u043d\u0430\u043c \u0441\u044e\u0434\u0430)<\/p>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/cb0\/0f6\/6d9\/cb00f66d90d45d5292f4f013991c76a0.png\" width=\"1600\" height=\"900\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/cb0\/0f6\/6d9\/cb00f66d90d45d5292f4f013991c76a0.png\"\/><\/figure>\n<p>\u0412\u043e\u0442 \u0438 \u0432\u0441\u0451 \ud83d\ude42<\/p>\n<p>\u0418\u043d\u0442\u0435\u0433\u0440\u0430\u0446\u0438\u044f \u043c\u0435\u0436\u0434\u0443 Dagster \u0438 Great Expectations \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442 \u0441\u043e\u0431\u043e\u0439 \u043c\u043e\u0449\u043d\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u044f \u043a\u0430\u0447\u0435\u0441\u0442\u0432\u0430 \u0434\u0430\u043d\u043d\u044b\u0445 \u0432 \u043a\u043e\u043d\u0432\u0435\u0439\u0435\u0440\u0430\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u0441 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c\u044e \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0438 \u0432\u0438\u0437\u0443\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439 \u043d\u0430 \u043a\u0430\u0436\u0434\u043e\u043c \u044d\u0442\u0430\u043f\u0435 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 \u0434\u0430\u043d\u043d\u044b\u0445.<\/p>\n<p>\u041d\u0430\u0434\u0435\u044e\u0441\u044c, \u044d\u0442\u043e\u0442 \u0442\u0443\u0442\u043e\u0440\u0438\u0430\u043b \u0431\u0443\u0434\u0435\u0442 \u043f\u043e\u043b\u0435\u0437\u0435\u043d \ud83d\ude42<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p> <!----> <!----><\/div>\n<p> <!----> <!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/746874\/\"> https:\/\/habr.com\/ru\/articles\/746874\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"\"><\/figure>\n<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u0440\u0442\u0435\u043c \u0428\u043d\u0430\u0439\u0434\u0435\u0440, \u0438 \u044f \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c DataScience \u0432 \u0411\u043b\u0430\u043d\u043a\u0435. \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u0432\u0430\u043c \u043e \u0442\u043e\u043c, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0434\u0432\u0430 \u043c\u043e\u0449\u043d\u044b\u0445 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430 \u2013 <a href=\"https:\/\/docs.dagster.io\/getting-started?_gl=1*1ut1d7v*_ga*MTgwMjM4NTc0OC4xNjc5MzgwNDM0*_ga_84VRQZG7TV*MTY4ODU5NzExMy45LjEuMTY4ODU5NzcwOC42MC4wLjA.\" rel=\"noopener noreferrer nofollow\"><strong>Dagster<\/strong><\/a> \u0438 <a href=\"https:\/\/docs.greatexpectations.io\/docs\/\" rel=\"noopener noreferrer nofollow\"><strong>Great Expectations<\/strong><\/a>.<\/p>\n<p>Great Expectations \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0438\u0442\u044c \u0442\u0430\u043a \u043d\u0430\u0437\u044b\u0432\u0430\u0435\u043c\u044b\u0435 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u044f \u043e\u0442 \u0432\u0430\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u0442\u043e \u0435\u0441\u0442\u044c \u0437\u0430\u0434\u0430\u0442\u044c \u043f\u0440\u0430\u0432\u0438\u043b\u0430 \u0438 \u0443\u0441\u043b\u043e\u0432\u0438\u044f, \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043e\u043b\u0436\u043d\u044b \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u043e\u0432\u0430\u0442\u044c.\u00a0<\/p>\n<p>Dagster, \u0441 \u0434\u0440\u0443\u0433\u043e\u0439 \u0441\u0442\u043e\u0440\u043e\u043d\u044b, \u044d\u0442\u043e \u043f\u043b\u0430\u0442\u0444\u043e\u0440\u043c\u0430 \u0441 \u043e\u0442\u043a\u0440\u044b\u0442\u044b\u043c \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u043c \u043a\u043e\u0434\u043e\u043c \u0434\u043b\u044f \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u043c\u0438, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u0432\u0430\u0442\u044c, \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0438 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u0442\u044c \u043f\u0430\u0439\u043f\u043b\u0430\u0439\u043d\u044b \u0434\u0430\u043d\u043d\u044b\u0445.\u00a0\u041d\u0430\u043f\u0438\u0441\u0430\u043d \u043d\u0430 python, \u0447\u0442\u043e \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f\u043c \u0433\u0438\u0431\u043a\u043e \u043d\u0430\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0442\u044c \u0438 \u0440\u0430\u0441\u0448\u0438\u0440\u044f\u0442\u044c \u0435\u0433\u043e \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u044c. <\/p>\n<p>\u0418\u0441\u0445\u043e\u0434\u043d\u044b\u0439 \u043a\u043e\u0434 \u043a \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043d\u0430 <a href=\"https:\/\/github.com\/TemaDobryyR\/great-expectations-with-dagster.git\" rel=\"noopener noreferrer nofollow\">GitHub<\/a>.<\/p>\n<p>\u0414\u0430\u0432\u0430\u0439\u0442\u0435 \u043d\u0430\u0447\u043d\u0435\u043c?)<\/p>\n<h2>\u041f\u043e\u0434\u0433\u043e\u0442\u043e\u0432\u043a\u0430<\/h2>\n<p>\u041d\u0430\u043c \u043f\u043e\u043d\u0430\u0434\u043e\u0431\u044f\u0442\u0441\u044f \u043a\u0430\u043a\u0438\u0435-\u043d\u0438\u0431\u0443\u0434\u044c \u0434\u0430\u043d\u043d\u044b\u0435, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u044b \u0437\u0430\u0433\u0440\u0443\u0437\u0438\u043c \u0432 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445. \u042f \u0440\u0435\u0448\u0438\u043b \u0432\u0437\u044f\u0442\u044c <a href=\"https:\/\/www.kaggle.com\/datasets\/vipin20\/transaction-data\" rel=\"noopener noreferrer nofollow\">\u043d\u0430\u0431\u043e\u0440 \u0434\u0430\u043d\u043d\u044b\u0445 \u0442\u0440\u0430\u043d\u0437\u0430\u043a\u0446\u0438\u0439<\/a> \u0441 Kaggle.<\/p>\n<p>\u0421\u043e\u0431\u0440\u0430\u043b \u0434\u043b\u044f \u0432\u0430\u0441 \u0433\u043e\u0442\u043e\u0432\u044b\u0439 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0439, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u043a\u043b\u043e\u043d\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043f\u043e \u044d\u0442\u043e\u0439 <a href=\"https:\/\/github.com\/TemaDobryyR\/simple-postgres-container.git\" rel=\"noopener noreferrer nofollow\"><u>\u0441\u0441\u044b\u043b\u043a\u0435<\/u><\/a>.\u00a0<\/p>\n<p>\u041f\u0435\u0440\u0435\u0439\u0434\u0435\u043c \u0432 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u044e <code>\/simple\u2011postgres\u2011container<\/code> \u0438 \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u043d\u0430\u0448\u0443 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<pre><code class=\"bash\">docker compose -f \"docker-compose.yaml\" up -d --build<\/code><\/pre>\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0443 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u043b\u043e\u043a\u0430\u043b\u044c\u043d\u044b\u0439 \u044d\u043a\u0437\u0435\u043c\u043f\u043b\u044f\u0440 \u0431\u0430\u0437\u044b \u0434\u0430\u043d\u043d\u044b\u0445 Postgres \u0441 \u0442\u0430\u0431\u043b\u0438\u0447\u043a\u043e\u0439 <em>transactions<\/em>.\u00a0<\/p>\n<p>\u0418\u0434\u0435\u043c \u0434\u0430\u043b\u044c\u0448\u0435)<\/p>\n<h2>1. Great Expectations \u0448\u0430\u0433 \u0437\u0430 \u0448\u0430\u0433\u043e\u043c<\/h2>\n<p>\u0412 \u044d\u0442\u043e\u0439 \u0447\u0430\u0441\u0442\u0438 \u043c\u044b \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u043c \u043f\u0430\u043a\u0435\u0442 <code>great-expectations<\/code>, \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u043c\u0441\u044f \u043a \u043d\u0430\u0448\u0435\u0439 \u0431\u0430\u0437\u0435 \u0434\u0430\u043d\u043d\u044b\u0445\u00a0 \u0438 \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043f\u0440\u043e\u0441\u0442\u043e\u0439 \u043d\u0430\u0431\u043e\u0440 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439.<\/p>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0443 <code>great-expectations<\/code>\u0438 \u0438\u043d\u0438\u0446\u0438\u0430\u043b\u0438\u0437\u0438\u0440\u0443\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442:<\/p>\n<pre><code class=\"bash\">pip install great-expectations great_expectations init<\/code><\/pre>\n<p>\u041a\u043e\u043c\u0430\u043d\u0434\u0430 \u0441\u043e\u0437\u0434\u0430\u0441\u0442 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u044e \u0441\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u0439 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439:<\/p>\n<p><code>great_expectations\/<br \/> \u251c\u2500\u2500 checkpoints<br \/> \u251c\u2500\u2500 expectations<br \/> \u251c\u2500\u2500 great_expectations.yml<br \/> \u251c\u2500\u2500 plugins<br \/> \u2502   \u2514\u2500\u2500 custom_data_docs<br \/> \u251c\u2500\u2500 profilers<br \/> \u2514\u2500\u2500 uncommitted<br \/> \u251c\u2500\u2500 config_variables.yml<br \/> \u251c\u2500\u2500 data_docs<br \/> \u2514\u2500\u2500 validations<\/code><\/p>\n<p>\u0414\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0438\u0435 \u043a\u043e\u043c\u0430\u043d\u0434\u044b \u0431\u0443\u0434\u0435\u043c \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u0438\u0437 \u0434\u0438\u0440\u0435\u043a\u0442\u043e\u0440\u0438\u0438 <code>\/great_expectations<\/code>:<\/p>\n<pre><code class=\"bash\">cd great_expectations<\/code><\/pre>\n<p>Great Expectations \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442\u00a0 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0438\u043d\u0442\u0435\u0440\u0444\u0435\u0439\u0441 \u043a\u043e\u043c\u0430\u043d\u0434\u043d\u043e\u0439 \u0441\u0442\u0440\u043e\u043a\u0438 \u0434\u043b\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0433\u043e \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u0445 \u0431\u043b\u043e\u043a\u043d\u043e\u0442\u043e\u0432 Jupyter, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u0440\u043e\u0432\u0435\u0434\u0443\u0442 \u0447\u0435\u0440\u0435\u0437 \u043a\u043e\u043d\u043a\u0440\u0435\u0442\u043d\u044b\u0435 \u044d\u0442\u0430\u043f\u044b \u0440\u0430\u0431\u043e\u0442\u044b \u0441 Great Expectations.<\/p>\n<h2>1.1 \u041d\u0430\u0441\u0442\u0440\u043e\u0438\u043c \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a \u0434\u0430\u043d\u043d\u044b\u0445<\/h2>\n<p>\u0412\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u0443:<\/p>\n<pre><code class=\"bash\">great_expectations datasource new<\/code><\/pre>\n<p>GE \u043f\u0440\u0435\u0434\u043b\u043e\u0436\u0438\u0442 \u0432\u0430\u0440\u0438\u0430\u043d\u0442\u044b \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f \u043a \u0434\u0430\u043d\u043d\u044b\u043c, \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u043c <code>Relational database (SQL)<\/code>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p><strong>P.S.<\/strong> \u041c\u043e\u0436\u0435\u0442 \u043f\u043e\u0442\u0440\u0435\u0431\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u043f\u0430\u043a\u0435\u0442\u043e\u0432 &#8212; \u0438\u0445 GE \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442 \u0441\u0430\u043c \u0432 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435.<\/p>\n<p>\u0414\u0430\u043b\u0435\u0435 \u0432\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u043d\u0430\u0448\u0443 \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>GE \u0441\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u0443\u0435\u0442 Jupyter \u043d\u043e\u0443\u0442\u0431\u0443\u043a, \u0432 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u043c\u044b \u0434\u043e\u043b\u0436\u043d\u044b \u0437\u0430\u043f\u043e\u043b\u043d\u0438\u0442\u044c \u0443\u0447\u0435\u0442\u043d\u044b\u0435 \u0434\u0430\u043d\u043d\u044b\u0435 \u0434\u043b\u044f \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u044f:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0411\u043e\u043b\u044c\u0448\u0435 \u043d\u0438\u0447\u0435\u0433\u043e \u043c\u0435\u043d\u044f\u0442\u044c \u043d\u0435 \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f, \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u043c \u043d\u043e\u0443\u0442\u0431\u0443\u043a \u0434\u043e \u043a\u043e\u043d\u0446\u0430, \u0437\u0430\u043a\u0440\u044b\u0432\u0430\u0435\u043c \u0438 \u043f\u0440\u0435\u0440\u044b\u0432\u0430\u0435\u043c \u043a\u043e\u043c\u0430\u043d\u0434\u0443 \u0432 \u0442\u0435\u0440\u043c\u0438\u043d\u0430\u043b\u0435. \u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044e \u043d\u0430\u0448\u0435\u0433\u043e \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u0430 \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u043e\u0436\u043d\u043e \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0432 \u0444\u0430\u0439\u043b\u0435 <code>great_expectations.yaml.<\/code><\/p>\n<details class=\"spoiler\">\n<summary>great_expectations.yaml<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"yaml\"># Welcome to Great Expectations! Always know what to expect from your data. # # Here you can define datasources, batch kwargs generators, integrations and # more. This file is intended to be committed to your repo. For help with # configuration please: #   - Read our docs: https:\/\/docs.greatexpectations.io\/docs\/guides\/connecting_to_your_data\/connect_to_data_overview\/#2-configure-your-datasource #   - Join our slack channel: http:\/\/greatexpectations.io\/slack  # config_version refers to the syntactic version of this config file, and is used in maintaining backwards compatibility # It is auto-generated and usually does not need to be changed. config_version: 3.0  # Datasources tell Great Expectations where your data lives and how to get it. # You can use the CLI command `great_expectations datasource new` to help you # add a new datasource. Read more at https:\/\/docs.greatexpectations.io\/docs\/guides\/connecting_to_your_data\/connect_to_data_overview datasources:   my_datasource:     class_name: Datasource     module_name: great_expectations.datasource     data_connectors:       default_runtime_data_connector_name:         batch_identifiers:           - default_identifier_name         class_name: RuntimeDataConnector         module_name: great_expectations.datasource.data_connector       default_inferred_data_connector_name:         include_schema_name: true         class_name: InferredAssetSqlDataConnector         module_name: great_expectations.datasource.data_connector         introspection_directives:           schema_name: public       default_configured_data_connector_name:         class_name: ConfiguredAssetSqlDataConnector         assets:           transactions:             class_name: Asset             schema_name: public             module_name: great_expectations.datasource.data_connector.asset         module_name: great_expectations.datasource.data_connector     execution_engine:       class_name: SqlAlchemyExecutionEngine       credentials:         host: localhost         port: '5432'         username: postgres         password: postgres         database: postgres         drivername: postgresql       module_name: great_expectations.execution_engine  # This config file supports variable substitution which enables: 1) keeping # secrets out of source control &amp; 2) environment-based configuration changes # such as staging vs prod. # # When GX encounters substitution syntax (like `my_key: ${my_value}` or # `my_key: $my_value`) in the great_expectations.yml file, it will attempt # to replace the value of `my_key` with the value from an environment # variable `my_value` or a corresponding key read from this config file, # which is defined through the `config_variables_file_path`. # Environment variables take precedence over variables defined here. # # Substitution values defined here can be a simple (non-nested) value, # nested value such as a dictionary, or an environment variable (i.e. ${ENV_VAR}) # # # https:\/\/docs.greatexpectations.io\/docs\/guides\/setup\/configuring_data_contexts\/how_to_configure_credentials   config_variables_file_path: uncommitted\/config_variables.yml  # The plugins_directory will be added to your python path for custom modules # used to override and extend Great Expectations. plugins_directory: plugins\/  stores: # Stores are configurable places to store things like Expectations, Validations # Data Docs, and more. These are for advanced users only - most users can simply # leave this section alone. # # Three stores are required: expectations, validations, and # evaluation_parameters, and must exist with a valid store entry. Additional # stores can be configured for uses such as data_docs, etc.   expectations_store:     class_name: ExpectationsStore     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: expectations\/    validations_store:     class_name: ValidationsStore     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: uncommitted\/validations\/    evaluation_parameter_store:     class_name: EvaluationParameterStore   checkpoint_store:     class_name: CheckpointStore     store_backend:       class_name: TupleFilesystemStoreBackend       suppress_store_backend_id: true       base_directory: checkpoints\/    profiler_store:     class_name: ProfilerStore     store_backend:       class_name: TupleFilesystemStoreBackend       suppress_store_backend_id: true       base_directory: profilers\/  expectations_store_name: expectations_store validations_store_name: validations_store evaluation_parameter_store_name: evaluation_parameter_store checkpoint_store_name: checkpoint_store  data_docs_sites:   # Data Docs make it simple to visualize data quality in your project. These   # include Expectations, Validations &amp; Profiles. The are built for all   # Datasources from JSON artifacts in the local repo including validations &amp;   # profiles from the uncommitted directory. Read more at https:\/\/docs.greatexpectations.io\/docs\/terms\/data_docs   local_site:     class_name: SiteBuilder     show_how_to_buttons: true     store_backend:       class_name: TupleFilesystemStoreBackend       base_directory: uncommitted\/data_docs\/local_site\/     site_index_builder:       class_name: DefaultSiteIndexBuilder  anonymous_usage_statistics:   enabled: true   data_context_id: ffd790a7-4454-4b53-b9b1-428bfcfb4e64 notebooks: include_rendered_content:   globally: false   expectation_validation_result: false   expectation_suite: false <\/code><\/pre>\n<\/p>\n<\/div>\n<\/details>\n<h2>1.2 \u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043d\u0430\u0431\u043e\u0440 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439<\/h2>\n<p>\u0412\u043e\u0437\u0432\u0440\u0430\u0449\u0430\u0435\u043c\u0441\u044f \u0432 \u043a\u043e\u043c\u0430\u043d\u0434\u0443\u044e \u0441\u0442\u0440\u043e\u043a\u0443 \u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u043c:<\/p>\n<pre><code class=\"bash\">great_expectations suite new<\/code><\/pre>\n<p>GE \u043d\u0430 \u0432\u044b\u0431\u043e\u0440 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u0442 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0441\u043f\u043e\u0441\u043e\u0431\u043e\u0432 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043d\u0430\u0431\u043e\u0440\u043e\u0432 \u043f\u0440\u043e\u0432\u0435\u0440\u043e\u043a. \u0412 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438 \u0440\u0435\u043a\u043e\u043c\u0435\u043d\u0434\u0443\u0435\u0442\u0441\u044f \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0441 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435\u043c <a href=\"https:\/\/docs.greatexpectations.io\/docs\/terms\/data_assistant\/\" rel=\"noopener noreferrer nofollow\">Data Assistant<\/a> \u2014 \u043f\u0440\u0435\u0434\u0432\u0430\u0440\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u043e\u0439 \u0443\u0442\u0438\u043b\u0438\u0442\u044b, \u0443\u043f\u0440\u043e\u0449\u0430\u044e\u0449\u0435\u0439 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439.<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0422\u0430\u043a\u0436\u0435 GE \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442 \u0442\u0440\u0438 \u0442\u0438\u043f\u0430 <code>DataConnector<\/code> \u043a\u043b\u0430\u0441\u0441\u043e\u0432. \u041f\u0440\u043e \u0432\u044b\u0431\u043e\u0440 \u043a\u043e\u043d\u043d\u0435\u043a\u0442\u043e\u0440\u043e\u0432 \u043c\u043e\u0436\u043d\u043e \u043f\u043e\u0447\u0438\u0442\u0430\u0442\u044c <a href=\"https:\/\/docs.greatexpectations.io\/docs\/0.15.50\/guides\/connecting_to_your_data\/how_to_choose_which_dataconnector_to_use\/\" rel=\"noopener noreferrer nofollow\"><u>\u0437\u0434\u0435\u0441\u044c<\/u><\/a>, \u0430 \u043d\u0430\u043c \u043d\u0443\u0436\u0435\u043d <code>default_configured_data_connector_name<\/code>:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u0412\u044b\u0431\u0438\u0440\u0430\u0435\u043c \u0430\u0441\u0441\u0435\u0442 \u0434\u0430\u043d\u043d\u044b\u0445 \u0438 \u0437\u0430\u0434\u0430\u0435\u043c \u0438\u043c\u044f \u043d\u0430\u0448\u0435\u043c\u0443 \u043d\u0430\u0431\u043e\u0440\u0443 \u043f\u0440\u043e\u0432\u0435\u0440\u043e\u043a:<\/p>\n<figure class=\"full-width\"><\/figure>\n<p>\u041d\u0430\u0431\u043e\u0440\u044b \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439, \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u044b\u0435 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043d\u043e\u0443\u0442\u0431\u0443\u043a\u0430, \u043d\u0435 \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u044b \u0434\u043b\u044f \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u0439 \u0441\u0440\u0435\u0434\u0435. \u041c\u044b \u043c\u043e\u0436\u0435\u043c \u043e\u0442\u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u0441\u0433\u0435\u043d\u0435\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b<code>\/expectations\/transactions\/suite.json<\/code> \u0438 \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0442\u0443\u0434\u0430 \u0435\u0449\u0435 \u043f\u0430\u0440\u0443 \u043e\u0436\u0438\u0434\u0430\u043d\u0438\u0439: \u043f\u0440\u043e\u0432\u0435\u0440\u0438\u043c, \u0447\u0442\u043e \u0432\u0441\u0435 \u0442\u0440\u0430\u043d\u0437\u0430\u043a\u0446\u0438\u0438 \u0438\u043c\u0435\u044e\u0442 ID, \u0430 \u043a\u043e\u043b\u0438\u0447\u0435\u0441\u0442\u0432\u043e \u043a\u0443\u043f\u043b\u0435\u043d\u043d\u044b\u0445 \u0442\u043e\u0432\u0430\u0440\u043e\u0432 \u0431\u043e\u043b\u044c\u0448\u0435 0. <\/p>\n<details class=\"spoiler\">\n<summary>\u0418\u0442\u043e\u0433\u043e\u0432\u044b\u0439 suite.json<\/summary>\n<div class=\"spoiler__content\">\n<pre><code class=\"yaml\">{   \"data_asset_type\": null,   \"expectation_suite_name\": \"transactions.suite\",   \"expectations\": [     {       \"expectation_type\": \"expect_table_row_count_to_be_between\",       \"kwargs\": {         \"max_value\": 2000000,         \"min_value\": 0       },       \"meta\": {         \"profiler_details\": {           \"metric_configuration\": {             \"domain_kwargs\": {},             \"metric_name\": \"table.row_count\",             \"metric_value_kwargs\": null           },           \"num_batches\": 1         }       }     },     {       \"expectation_type\": \"expect_table_columns_to_match_set\",       \"kwargs\": {         \"column_set\": [           \"ItemCode\",           \"UserId\",           \"NumberOfItemsPurchased\",           \"Country\",           \"TransactionTime\",           \"ItemDescription\",           \"TransactionId\",           \"CostPerItem\"         ],         \"exact_match\": null       },       \"meta\": {         \"profiler_details\": {           \"success_ratio\": 1.0         }       }     },     {       \"expectation_type\": \"expect_column_values_to_not_be_null\",       \"kwargs\": {         \"column\": \"TransactionId\"       }     },     {       \"expectation_type\":<\/code><\/pre>\n<\/div>\n<\/details>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-349999","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/349999","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=349999"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/349999\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=349999"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=349999"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=349999"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}