{"id":473602,"date":"2025-09-02T17:03:39","date_gmt":"2025-09-02T17:03:39","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=473602"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=473602","title":{"rendered":"<span>We\u2019ve learned how to migrate databases from Oracle to Postgres Pro at 41 TB\/day<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/3d4\/dd7\/d14\/3d4dd7d14069882c71f6279e12231fd1.png\" width=\"3250\" height=\"1833\" sizes=\"auto, (max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/getpro\/habr\/upload_files\/3d4\/dd7\/d14\/3d4dd7d14069882c71f6279e12231fd1.png 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/3d4\/dd7\/d14\/3d4dd7d14069882c71f6279e12231fd1.png 781w\" loading=\"lazy\" decode=\"async\"\/><\/figure>\n<p>Imagine a typical scenario: your company runs on Oracle Database, and over the years, you\u2019ve accumulated tens of terabytes of data. Now you want to move to Postgres Pro \u2014 but the migration process quickly turns into a nightmare:<\/p>\n<ul>\n<li>\n<p>It takes forever\u00a0to move all the data.<\/p>\n<\/li>\n<li>\n<p>The source system can\u2019t stop\u00a0\u2014 it has to keep running, even though the data is constantly changing.<\/p>\n<\/li>\n<li>\n<p>There\u2019s a real risk of data loss or corruption\u00a0\u2014 from transmission errors to type mismatches and other gremlins.<\/p>\n<\/li>\n<\/ul>\n<p>We decided to tackle these problems head-on and created\u00a0ProGate, a toolkit that makes life a lot easier for DBAs and speeds up the migration to Postgres Pro.<\/p>\n<figure class=\"full-width\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/3c0\/420\/f37\/3c0420f37d5f4bc2961bd1a86e8d7f10.png\" width=\"1560\" height=\"920\" sizes=\"auto, (max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/getpro\/habr\/upload_files\/3c0\/420\/f37\/3c0420f37d5f4bc2961bd1a86e8d7f10.png 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/3c0\/420\/f37\/3c0420f37d5f4bc2961bd1a86e8d7f10.png 781w\" loading=\"lazy\" decode=\"async\"\/><\/figure>\n<h3>What\u2019s ProGate?<\/h3>\n<p>ProGate is an all-in-one solution for moving data into Postgres Pro. It\u2019s a set of specialized tools that cover\u00a0every step\u00a0of the migration process:<\/p>\n<ul>\n<li>\n<p>ProCopy\u00a0\u2014 high-speed initial data load.<\/p>\n<\/li>\n<li>\n<p>ProSync\u00a0\u2014 continuous change synchronization (CDC \u2014 Change Data Capture).<\/p>\n<\/li>\n<li>\n<p>ProCheck\u00a0\u2014 post-migration data quality and integrity checks.<\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s go through them one by one.<\/p>\n<h3>Step 1. Initial data load with ProCopy<\/h3>\n<p>ProCopy\u00a0is a console utility for blasting large amounts of data into Postgres Pro as fast as possible \u2014 without having to stop the source system. That means minimal downtime and minimal impact on running applications.<\/p>\n<p>In our recent synthetic tests, we reached\u00a0200\u2013500 MB\/sec\u00a0for Oracle \u2192 Postgres Pro migrations, which works out to about\u00a041 TB\/day. For PostgreSQL \u2192 Postgres Pro, we hit around\u00a01 GB\/sec.<\/p>\n<h4>How does it work<\/h4>\n<p>ProCopy is written in\u00a0Go, so it\u2019s lightweight, highly concurrent, and ready to chew through databases of any size. Under the hood, it uses:<\/p>\n<ul>\n<li>\n<p>A pool of parallel read\/write processes.<\/p>\n<\/li>\n<li>\n<p>An internal data bus for fast process communication.<\/p>\n<\/li>\n<li>\n<p>Robust error handling with retries for problematic records.<\/p>\n<\/li>\n<\/ul>\n<h4>ProCopy use case<\/h4>\n<p>Say you\u2019ve got a\u00a0CUSTOMERS\u00a0table in Oracle with billions of rows. With ProCopy, you can:<\/p>\n<ul>\n<li>\n<p>Run the migration in parallel, tuning the number of reader\/writer processes and batch sizes.<\/p>\n<\/li>\n<li>\n<p>Exclude unnecessary columns.<\/p>\n<\/li>\n<li>\n<p>Rename columns or change their data types.<\/p>\n<\/li>\n<li>\n<p>Transform\u00a0<code>NULL<\/code>s into default values\u00a0on the fly.<\/p>\n<\/li>\n<\/ul>\n<h4>Highlights<\/h4>\n<ul>\n<li>\n<p>Blazing fast data transfer.<\/p>\n<\/li>\n<li>\n<p>Flexible configuration (YAML\/JSON).<\/p>\n<\/li>\n<li>\n<p>Resume from where you left off.<\/p>\n<\/li>\n<li>\n<p>Support for complex data types (LOB, XML, JSON, etc.).<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/a19\/2f4\/06d\/a192f406d4515755f4771386a63df0ac.png\" width=\"1560\" height=\"920\" sizes=\"auto, (max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/getpro\/habr\/upload_files\/a19\/2f4\/06d\/a192f406d4515755f4771386a63df0ac.png 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/a19\/2f4\/06d\/a192f406d4515755f4771386a63df0ac.png 781w\" loading=\"lazy\" decode=\"async\"\/><\/figure>\n<h3>Step 2. Change synchronization with ProSync (CDC)<\/h3>\n<p>ProSync\u00a0keeps Oracle and Postgres Pro in sync by continuously capturing changes (CDC) and applying them to the target database. This makes it possible to migrate with\u00a0very short downtime.<\/p>\n<h4>How does it work<\/h4>\n<p>Oracle stores every data change \u2014 inserts, updates, deletes \u2014 in\u00a0redo logs. ProSync tails these logs in real time, detects each change, and immediately applies it to Postgres Pro.<\/p>\n<p>As a result, your Postgres Pro instance is always up-to-date with Oracle, down to the last committed transaction.<\/p>\n<h4>Real-world example<\/h4>\n<p>A bank can\u2019t afford even an hour of downtime. With ProSync, they can migrate data and apps gradually, while the old Oracle system stays live. Customers keep working with Oracle, and every transaction is instantly mirrored to Postgres Pro. Once everything\u2019s in sync, switching over takes minutes.<\/p>\n<h4>Highlights<\/h4>\n<ul>\n<li>\n<p>Minimal load on the source DB.<\/p>\n<\/li>\n<li>\n<p>Reliable error handling and replication health monitoring.<\/p>\n<\/li>\n<\/ul>\n<h3>Step 3. Data quality check with ProCheck<\/h3>\n<p>ProCheck\u00a0verifies that the migration went smoothly and that the data in Postgres Pro is\u00a0exactly\u00a0the same as in Oracle.<\/p>\n<h4>How does it work<\/h4>\n<p>It compares tables, rows, and columns between the two databases. It catches discrepancies, conversion errors, and missing data \u2014 and gives you a detailed report.<\/p>\n<h4>ProGate use case<\/h4>\n<p>After migrating a financial system, you want to make sure every balance matches to the last cent. ProCheck will go row-by-row to confirm that nothing\u2019s been lost or altered.<\/p>\n<p>ProGate is perfect for:<\/p>\n<ul>\n<li>\n<p>Huge databases\u00a0(terabytes and up).<\/p>\n<\/li>\n<li>\n<p>Hot migrations\u00a0with minimal downtime.<\/p>\n<\/li>\n<li>\n<p>Strict data consistency requirements.<\/p>\n<\/li>\n<\/ul>\n<h3>Known Limitations<\/h3>\n<p>While ProGate solves a lot of headaches, you should keep in mind:<\/p>\n<ul>\n<li>\n<p>Schema changes during migration may require manual handling.<\/p>\n<\/li>\n<li>\n<p>Custom data types may need special mapping.<\/p>\n<\/li>\n<li>\n<p>Tables without primary keys \u2014 ProSync works best with unique identifiers.<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/98a\/156\/377\/98a156377ce7733e7b49e3d3ba55983e.png\" width=\"1560\" height=\"920\" sizes=\"auto, (max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/getpro\/habr\/upload_files\/98a\/156\/377\/98a156377ce7733e7b49e3d3ba55983e.png 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/98a\/156\/377\/98a156377ce7733e7b49e3d3ba55983e.png 781w\" loading=\"lazy\" decode=\"async\"\/><\/figure>\n<h3>What\u2019s next<\/h3>\n<p>The\u00a0public release\u00a0of ProGate is planned for this fall. Our roadmap includes:<\/p>\n<ul>\n<li>\n<p>A graphical UI, backend, and API.<\/p>\n<\/li>\n<li>\n<p>New sources\/targets (MS SQL Server, MySQL, Shardman).<\/p>\n<\/li>\n<\/ul>\n<p>In short: ProGate will let you move from Oracle to Postgres Pro\u00a0<em>fast<\/em>,\u00a0<em>safe<\/em>, and with zero drama \u2014 no endless downtime, no missing data, no nasty surprises.<\/p>\n<p>We\u2019ll share more about the technical internals after the public launch.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/936278\/\"> https:\/\/habr.com\/ru\/articles\/936278\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<figure class=\"full-width\"><\/figure>\n<p>Imagine a typical scenario: your company runs on Oracle Database, and over the years, you\u2019ve accumulated tens of terabytes of data. Now you want to move to Postgres Pro \u2014 but the migration process quickly turns into a nightmare:<\/p>\n<ul>\n<li>\n<p>It takes forever\u00a0to move all the data.<\/p>\n<\/li>\n<li>\n<p>The source system can\u2019t stop\u00a0\u2014 it has to keep running, even though the data is constantly changing.<\/p>\n<\/li>\n<li>\n<p>There\u2019s a real risk of data loss or corruption\u00a0\u2014 from transmission errors to type mismatches and other gremlins.<\/p>\n<\/li>\n<\/ul>\n<p>We decided to tackle these problems head-on and created\u00a0ProGate, a toolkit that makes life a lot easier for DBAs and speeds up the migration to Postgres Pro.<\/p>\n<figure class=\"full-width\"><\/figure>\n<h3>What\u2019s ProGate?<\/h3>\n<p>ProGate is an all-in-one solution for moving data into Postgres Pro. It\u2019s a set of specialized tools that cover\u00a0every step\u00a0of the migration process:<\/p>\n<ul>\n<li>\n<p>ProCopy\u00a0\u2014 high-speed initial data load.<\/p>\n<\/li>\n<li>\n<p>ProSync\u00a0\u2014 continuous change synchronization (CDC \u2014 Change Data Capture).<\/p>\n<\/li>\n<li>\n<p>ProCheck\u00a0\u2014 post-migration data quality and integrity checks.<\/p>\n<\/li>\n<\/ul>\n<p>Let\u2019s go through them one by one.<\/p>\n<h3>Step 1. Initial data load with ProCopy<\/h3>\n<p>ProCopy\u00a0is a console utility for blasting large amounts of data into Postgres Pro as fast as possible \u2014 without having to stop the source system. That means minimal downtime and minimal impact on running applications.<\/p>\n<p>In our recent synthetic tests, we reached\u00a0200\u2013500 MB\/sec\u00a0for Oracle \u2192 Postgres Pro migrations, which works out to about\u00a041 TB\/day. For PostgreSQL \u2192 Postgres Pro, we hit around\u00a01 GB\/sec.<\/p>\n<h4>How does it work<\/h4>\n<p>ProCopy is written in\u00a0Go, so it\u2019s lightweight, highly concurrent, and ready to chew through databases of any size. Under the hood, it uses:<\/p>\n<ul>\n<li>\n<p>A pool of parallel read\/write processes.<\/p>\n<\/li>\n<li>\n<p>An internal data bus for fast process communication.<\/p>\n<\/li>\n<li>\n<p>Robust error handling with retries for problematic records.<\/p>\n<\/li>\n<\/ul>\n<h4>ProCopy use case<\/h4>\n<p>Say you\u2019ve got a\u00a0CUSTOMERS\u00a0table in Oracle with billions of rows. With ProCopy, you can:<\/p>\n<ul>\n<li>\n<p>Run the migration in parallel, tuning the number of reader\/writer processes and batch sizes.<\/p>\n<\/li>\n<li>\n<p>Exclude unnecessary columns.<\/p>\n<\/li>\n<li>\n<p>Rename columns or change their data types.<\/p>\n<\/li>\n<li>\n<p>Transform\u00a0<code>NULL<\/code>s into default values\u00a0on the fly.<\/p>\n<\/li>\n<\/ul>\n<h4>Highlights<\/h4>\n<ul>\n<li>\n<p>Blazing fast data transfer.<\/p>\n<\/li>\n<li>\n<p>Flexible configuration (YAML\/JSON).<\/p>\n<\/li>\n<li>\n<p>Resume from where you left off.<\/p>\n<\/li>\n<li>\n<p>Support for complex data types (LOB, XML, JSON, etc.).<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><\/figure>\n<h3>Step 2. Change synchronization with ProSync (CDC)<\/h3>\n<p>ProSync\u00a0keeps Oracle and Postgres Pro in sync by continuously capturing changes (CDC) and applying them to the target database. This makes it possible to migrate with\u00a0very short downtime.<\/p>\n<h4>How does it work<\/h4>\n<p>Oracle stores every data change \u2014 inserts, updates, deletes \u2014 in\u00a0redo logs. ProSync tails these logs in real time, detects each change, and immediately applies it to Postgres Pro.<\/p>\n<p>As a result, your Postgres Pro instance is always up-to-date with Oracle, down to the last committed transaction.<\/p>\n<h4>Real-world example<\/h4>\n<p>A bank can\u2019t afford even an hour of downtime. With ProSync, they can migrate data and apps gradually, while the old Oracle system stays live. Customers keep working with Oracle, and every transaction is instantly mirrored to Postgres Pro. Once everything\u2019s in sync, switching over takes minutes.<\/p>\n<h4>Highlights<\/h4>\n<ul>\n<li>\n<p>Minimal load on the source DB.<\/p>\n<\/li>\n<li>\n<p>Reliable error handling and replication health monitoring.<\/p>\n<\/li>\n<\/ul>\n<h3>Step 3. Data quality check with ProCheck<\/h3>\n<p>ProCheck\u00a0verifies that the migration went smoothly and that the data in Postgres Pro is\u00a0exactly\u00a0the same as in Oracle.<\/p>\n<h4>How does it work<\/h4>\n<p>It compares tables, rows, and columns between the two databases. It catches discrepancies, conversion errors, and missing data \u2014 and gives you a detailed report.<\/p>\n<h4>ProGate use case<\/h4>\n<p>After migrating a financial system, you want to make sure every balance matches to the last cent. ProCheck will go row-by-row to confirm that nothing\u2019s been lost or altered.<\/p>\n<p>ProGate is perfect for:<\/p>\n<ul>\n<li>\n<p>Huge databases\u00a0(terabytes and up).<\/p>\n<\/li>\n<li>\n<p>Hot migrations\u00a0with minimal downtime.<\/p>\n<\/li>\n<li>\n<p>Strict data consistency requirements.<\/p>\n<\/li>\n<\/ul>\n<h3>Known Limitations<\/h3>\n<p>While ProGate solves a lot of headaches, you should keep in mind:<\/p>\n<ul>\n<li>\n<p>Schema changes during migration may require manual handling.<\/p>\n<\/li>\n<li>\n<p>Custom data types may need special mapping.<\/p>\n<\/li>\n<li>\n<p>Tables without primary keys \u2014 ProSync works best with unique identifiers.<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><\/figure>\n<h3>What\u2019s next<\/h3>\n<p>The\u00a0public release\u00a0of ProGate is planned for this fall. Our roadmap includes:<\/p>\n<ul>\n<li>\n<p>A graphical UI, backend, and API.<\/p>\n<\/li>\n<li>\n<p>New sources\/targets (MS SQL Server, MySQL, Shardman).<\/p>\n<\/li>\n<\/ul>\n<p>In short: ProGate will let you move from Oracle to Postgres Pro\u00a0<em>fast<\/em>,\u00a0<em>safe<\/em>, and with zero drama \u2014 no endless downtime, no missing data, no nasty surprises.<\/p>\n<p>We\u2019ll share more about the technical internals after the public launch.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/936278\/\"> https:\/\/habr.com\/ru\/articles\/936278\/<\/a><br \/><\/br><\/br><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-473602","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/473602","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=473602"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/473602\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=473602"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=473602"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=473602"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}