{"id":405683,"date":"2024-06-29T18:28:55","date_gmt":"2024-06-29T18:28:55","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=405683"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=405683","title":{"rendered":"<span>The Testcontainers\u2019 MongoDB Module and Spring Data MongoDB Reactive in Action<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-1\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h3 id=\"1-introduction\">1. Introduction<\/h3>\n<p>  <\/p>\n<p>How can I easily test my MongoDB multi-document transaction code without setting up MongoDB on my device? One might argue that they have to set it up first because in order to carry out such a transaction it needs a session which requires a replica set. Thankfully, there is no need to create a 3-node replica set and we can run these transactions only against a single database instance.<\/p>\n<p><a name=\"habracut\"><\/a>  <\/p>\n<p>To achieve this, we may do the following:<\/p>\n<p>  <\/p>\n<ul>\n<li>Run a MongoDB container of version 4 or higher and specify a &#8212;replSet command;<\/li>\n<li>Initialize a single replica set by executing a proper command;<\/li>\n<li>Wait for the initialization to complete; <\/li>\n<li>Connect to a standalone without specifying a replica set in order not to worry about modifying our OS host file.<\/li>\n<\/ul>\n<p>  <\/p>\n<p>It is worth mentioning that a replica set is not the only option here because MongoDB version 4.2 introduces distributed transactions in sharded clusters, which is beyond the scope of this article.<\/p>\n<p>  <\/p>\n<p>There are a lot of ways of how to initialize a replica set, including Docker compose, bash scripts, services in a CI\/CD etc. However, it takes some extra work in terms of scripting, handling random ports, and making it part of the CI\/CD process. Fortunately, starting from Testcontainers\u2019 version 1.14.2 we are able to delegate all the heavy lifting to <a href=\"https:\/\/www.testcontainers.org\/modules\/databases\/mongodb\/\" rel=\"nofollow\">the MongoDB Module<\/a>.<\/p>\n<p>  <\/p>\n<p>Let us try it out on a small warehouse management system based on Spring Boot 2.3. In the recent past one had to use <code>ReactiveMongoOperations<\/code> and its <code>inTransaction<\/code> method, but since Spring Data MongoDB 2.2 M4 we have been able to leverage the good old <code>@Transactional<\/code> annotation or more advanced <code>TransactionalOperator<\/code>.<\/p>\n<p>  <\/p>\n<p>Our application should have a REST API to provide the information on successfully processed files including the number of the documents modified. Regarding the files causing errors along the way, we should skip them to process all the files.<\/p>\n<p>  <\/p>\n<p>It may be noted that even though duplicated articles and their sizes within a single file are a rare case, this possibility is quite realistic, and therefore should be handled as well.<\/p>\n<p>  <\/p>\n<p>As per business requirements to our system, we already have some products in our database and we upload a bunch of Excel (xlsx) files to update some fields of the matched documents in our storage. Data is supposed to be only at the first sheet of any workbook. Each file is processed in a separate multi-document transaction to prevent simultaneous modifications of the same documents. For example, Figure 1 shows collision cases on how a transaction ends up except for a possible scenario when transactions are executed sequentially (json representation is shortened here for the sake of simplicity). Transactional behavior helps us to avoid clashing the data and guarantees consistency.<\/p>\n<p>  <\/p>\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/4a6\/549\/039\/4a65490391b07ab974ee0635b3ee19e2.png\" alt=\"Figure 1\" title=\"Transaction sequence diagram: collision cases\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/4a6\/549\/039\/4a65490391b07ab974ee0635b3ee19e2.png\"\/><br \/>  <em>Figure 1 Transaction sequence diagram: collision cases<\/em><\/p>\n<p>  <\/p>\n<p>As for a product collection, we have an article as a unique index. At the same time, each article is bound to a concrete size. Therefore, it is important for our application to verify that both of them are in the database before updating. Figure 2 gives an insight into this collection.<\/p>\n<p>  <\/p>\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/81a\/ecf\/fd7\/81aecffd7e7a632365d43a4d9792ddf9.png\" alt=\"Figure 2\" title=\"Product collection details\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/81a\/ecf\/fd7\/81aecffd7e7a632365d43a4d9792ddf9.png\"\/><br \/>  <em>Figure 2 Product collection details<\/em><\/p>\n<p>  <\/p>\n<h3 id=\"2-business-logic-implementation\">2. Business logic implementation<\/h3>\n<p>  <\/p>\n<p>Let us elaborate on the major points of the above-mentioned business logic and start with <code>ProductController<\/code> as an entry point for the processing. You can find a complete project on <a href=\"https:\/\/github.com\/silaev\/wms\" rel=\"nofollow\">GitHub<\/a>. Prerequisites are Java8+ and Docker.<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@PatchMapping(   consumes = MediaType.MULTIPART_FORM_DATA_VALUE,   produces = MediaType.APPLICATION_STREAM_JSON_VALUE ) public ResponseEntity&lt;Flux&lt;FileUploadDto>> patchProductQuantity(   @RequestPart(\"file\") Flux&lt;FilePart> files,   @AuthenticationPrincipal Principal principal ) {   log.debug(\"shouldPatchProductQuantity\");   return ResponseEntity.accepted().body(     uploadProductService.patchProductQuantity(files, principal.getName())   ); }<\/code><\/pre>\n<p>  <\/p>\n<p>1) Wrap a response in a <code>ResponseEntity<\/code> and return the <code>flux<\/code> of the <code>FileUploadDto<\/code>;<br \/>  2) Get a current authentication principal, coming in handy later on;<br \/>  3) Pass the <code>flux<\/code> of the <code>FilePart<\/code> to process.<\/p>\n<p>  <\/p>\n<p>Here is the <code>patchProductQuantity<\/code> method of the <code>UploadProductServiceImpl<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">public Flux&lt;FileUploadDto> patchProductQuantity(   final Flux&lt;FilePart> files,   final String userName ) {   return Mono.fromRunnable(() -> initRootDirectory(userName))     .publishOn(Schedulers.newBoundedElastic(1, 1, \"initRootDirectory\"))     .log(String.format(\"cleaning-up directory: %s\", userName))     .thenMany(files.flatMap(f ->         saveFileToDiskAndUpdate(f, userName)           .subscribeOn(Schedulers.boundedElastic())       )     ); }<\/code><\/pre>\n<p>  <\/p>\n<p>1) Use the name of the user as the root directory name;<br \/>  2) Do the blocking initialization of the root directory on a separate elastic thread;<br \/>  3) For each Excel file:<br \/>  3.1) Save it on a disk;<br \/>  3.2) Then update the quantity of the products on a separate elastic thread, as blocking processing of the file is ran.<\/p>\n<p>  <\/p>\n<p>The <code>saveFileToDiskAndUpdate<\/code> method does the following logic:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private Mono&lt;FileUploadDto> saveFileToDiskAndUpdate(   final FilePart file,   final String userName ) {   final String fileName = file.filename();   final Path path = Paths.get(pathToStorage, userName, fileName);   return Mono.just(path)     .log(String.format(\"A file: %s has been uploaded\", fileName))     .flatMap(file::transferTo)     .log(String.format(\"A file: %s has been saved\", fileName))     .then(processExcelFile(fileName, userName, path)); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>Copy the content of the file to the user\u2019s directory;<\/li>\n<li>After the copy stage is completed, call the <code>processExcelFile<\/code> method.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>At this point, we are going to divide logic in accordance with the size of the file:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private Mono&lt;FileUploadDto> processExcelFile(   final String fileName,   final String userName,   final Path path ) {   return Mono.fromCallable(() -> Files.size(path))     .flatMap(size -> {       if (size >= bigFileSizeThreshold) {         return processBigExcelFile(fileName, userName);       } else {         return processSmallExcelFile(fileName, userName);       }     }); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>Wrap the blocking <code>Files.size(path)<\/code> call in <code>Mono.fromCallable<\/code>;<\/li>\n<li><code>bigFileSizeThreshold<\/code> is injected from a proper application.yml file via <code>@Value(\"${upload-file.bigFileSizeThreshold}\")<\/code>.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Before going into detail on processing Excel files depending on their size, we should take a look at the <code>getProducts<\/code> method of the <code>ExcelFileDaoImpl<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@Override public Flux&lt;Product> getProducts(   final String pathToStorage,   final String fileName,   final String userName ) {   return Flux.defer(() -> {     FileInputStream is;     Workbook workbook;     try {       final File file = Paths.get(pathToStorage, userName, fileName).toFile();       verifyFileAttributes(file);       is = new FileInputStream(file);       workbook = StreamingReader.builder()         .rowCacheSize(ROW_CACHE_SIZE)         .bufferSize(BUFFER_SIZE)         .open(is);     } catch (IOException e) {       return Mono.error(new UploadProductException(         String.format(\"An exception has been occurred while parsing a file: %s \" +           \"has been saved\", fileName), e));     }      try {       final Sheet datatypeSheet = workbook.getSheetAt(0);       final Iterator&lt;Row> iterator = datatypeSheet.iterator();        final AtomicInteger rowCounter = new AtomicInteger();       if (iterator.hasNext()) {         final Row currentRow = iterator.next();         rowCounter.incrementAndGet();         verifyExcelFileHeader(fileName, currentRow);       }       return Flux.&lt;Product>create(fluxSink -> fluxSink.onRequest(value -> {         try {           for (int i = 0; i &lt; value; i++) {             if (!iterator.hasNext()) {               fluxSink.complete();               return;             }              final Row currentRow = iterator.next();             final Product product = Objects.requireNonNull(getProduct(               FileRow.builder()                 .fileName(fileName)                 .currentRow(currentRow)                 .rowCounter(rowCounter.incrementAndGet())                 .build()             ), \"product is not supposed to be null\");             fluxSink.next(product);           }         } catch (Exception e1) {           fluxSink.error(e1);         }       })).doFinally(signalType -> {         try {           is.close();           workbook.close();         } catch (IOException e1) {           log.error(\"Error has occurred while releasing {} resources: {}\", fileName, e1);         }       });     } catch (Exception e) {       return Mono.error(e);     }   }); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li><code>differ<\/code> the whole logic once there is a new subscriber;<\/li>\n<li>Verify the excel file header;<\/li>\n<li>Create a <code>flux<\/code> to provide the requested number of products;<\/li>\n<li>Convert an Excel row into a <code>Product<\/code> domain object;<\/li>\n<li>Finally, close all of the opened resources.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Getting back to the processing of the Excel files in the <code>UploadProductServiceImpl<\/code>, we are going to use the MongoDB\u2019s <code>bulkWrite<\/code> method on a collection to update products in bulk, which requires the eagerly evaluated list of the <code>UpdateOneModel<\/code>. In practice, collecting such a list is a memory-consuming operation, especially for big files.<\/p>\n<p>  <\/p>\n<p>Regarding small Excel files, we provide a more detailed log and do additional validation check:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private Mono&lt;FileUploadDto> processSmallExcelFile(   final String fileName,   final String userName ) {   log.debug(\"processSmallExcelFile: {}\", fileName);   return excelFileDao.getProducts(pathToStorage, fileName, userName)     .reduce(new ConcurrentHashMap&lt;ProductArticleSizeDto, Tuple2&lt;UpdateOneModel&lt;Document>, BigInteger>>(),       (indexMap, product) -> {         final BigInteger quantity = product.getQuantity();         indexMap.merge(           new ProductArticleSizeDto(product.getArticle(), product.getSize()),           Tuples.of(             updateOneModelConverter.convert(Tuples.of(product, quantity, userName)),             quantity           ),           (oldValue, newValue) -> {             final BigInteger mergedQuantity = oldValue.getT2().add(newValue.getT2());             return Tuples.of(               updateOneModelConverter.convert(Tuples.of(product, mergedQuantity, userName)),               mergedQuantity             );           }          );         return indexMap;       })     .filterWhen(productIndexFile ->       productDao.findByArticleIn(extractArticles(productIndexFile.keySet()))         .&lt;ProductArticleSizeDto>handle(           (productArticleSizeDto, synchronousSink) -> {             if (productIndexFile.containsKey(productArticleSizeDto)) {               synchronousSink.next(productArticleSizeDto);             } else {               synchronousSink.error(new UploadProductException(                 String.format(                   \"A file %s does not have an article: %d with size: %s\",                   fileName,                   productArticleSizeDto.getArticle(),                   productArticleSizeDto.getSize()                 )               ));             }           })         .count()         .handle((sizeDb, synchronousSink) -> {           final int sizeFile = productIndexFile.size();           if (sizeDb == sizeFile) {             synchronousSink.next(Boolean.TRUE);           } else {             synchronousSink.error(new UploadProductException(               String.format(                 \"Inconsistency between total element size in MongoDB: %d and a file %s: %d\",                 sizeDb,                 fileName,                 sizeFile               )             ));           }         })     ).onErrorResume(e -> {       log.debug(\"Exception while processExcelFile fileName: {}: {}\", fileName, e);       return Mono.empty();     }).flatMap(productIndexFile ->       productPatcherService.incrementProductQuantity(         fileName,         productIndexFile.values().stream().map(Tuple2::getT1).collect(Collectors.toList()),         userName       )     ).map(bulkWriteResult -> FileUploadDto.builder()       .fileName(fileName)       .matchedCount(bulkWriteResult.getMatchedCount())       .modifiedCount(bulkWriteResult.getModifiedCount())       .build()     ); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li><code>reduce<\/code> helps us handle duplicate products whose quantities should be summed up; <\/li>\n<li>Collect a map to get the list of the <code>ProductArticleSizeDto<\/code> against the pair of the list of the <code>UpdateOneModel<\/code> and the total quantity for a product. The former is in use for matching an article and its size in the file with those that are in the database via a projection <code>ProductArticleSizeDto<\/code>;<\/li>\n<li>Use the atomic <code>merge<\/code> method of the <code>ConcurrentMap<\/code> to sum up the quantity of the same products and create a new <code>UpdateOneModel<\/code>;<\/li>\n<li>Filter out all products in the file by those product\u2019s articles that are in the database;<\/li>\n<li>Each <code>ProductArticleSizeDto<\/code> found in the storage matches a <code>ProductArticleSizeDto<\/code> from the file summed up by quantity;<\/li>\n<li>Then <code>count<\/code> the result after filtration which should be equal to the distinct number of products in the file;<\/li>\n<li>Use the <code>onErrorResume<\/code> method to continue when any error occurs because we need to process all files as mentioned in the requirements;<\/li>\n<li>Extract the list of the <code>UpdateOneModel<\/code> from the map collected earlier to be further used in the <code>incrementProductQuantity<\/code> method;<\/li>\n<li>Then run the <code>incrementProductQuantity<\/code> method as a sub-process within <code>flatMap<\/code> and <code>map<\/code> its result in <code>FileUploadDto<\/code> that our business users are in need of.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Even though the <code>filterWhen<\/code> and the subsequent <code>productDao.findByArticleIn<\/code> allow us to do some additional validation at an early stage, they come at a price, which is especially noticeable while processing big files in practice. However, the <code>incrementProductQuantity<\/code> method can compare the number of modified documents and match them against the number of the distinct products in the file. Knowing that, we can implement a more light-weight option to process big files:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private Mono&lt;FileUploadDto> processBigExcelFile(   final String fileName,   final String userName ) {   log.debug(\"processBigExcelFile: {}\", fileName);   return excelFileDao.getProducts(pathToStorage, fileName, userName)     .reduce(new ConcurrentHashMap&lt;Product, Tuple2&lt;UpdateOneModel&lt;Document>, BigInteger>>(),       (indexMap, product) -> {         final BigInteger quantity = product.getQuantity();         indexMap.merge(           product,           Tuples.of(             updateOneModelConverter.convert(Tuples.of(product, quantity, userName)),             quantity           ),           (oldValue, newValue) -> {             final BigInteger mergedQuantity = oldValue.getT2().add(newValue.getT2());             return Tuples.of(               updateOneModelConverter.convert(Tuples.of(product, mergedQuantity, userName)),               mergedQuantity             );           }          );         return indexMap;       })     .map(indexMap -> indexMap.values().stream().map(Tuple2::getT1).collect(Collectors.toList()))     .onErrorResume(e -> {       log.debug(\"Exception while processExcelFile: {}: {}\", fileName, e);       return Mono.empty();     }).flatMap(dtoList ->       productPatcherService.incrementProductQuantity(         fileName,         dtoList,         userName       )     ).map(bulkWriteResult -> FileUploadDto.builder()       .fileName(fileName)       .matchedCount(bulkWriteResult.getMatchedCount())       .modifiedCount(bulkWriteResult.getModifiedCount())       .build()     ); }<\/code><\/pre>\n<p>  <\/p>\n<p>Here is the <code>ProductAndUserNameToUpdateOneModelConverter<\/code> that we have used to create an <code>UpdateOneModel<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@Component public class ProductAndUserNameToUpdateOneModelConverter implements   Converter&lt;Tuple3&lt;Product, BigInteger, String>, UpdateOneModel&lt;Document>> {    @Override   @NonNull   public UpdateOneModel&lt;Document> convert(@NonNull Tuple3&lt;Product, BigInteger, String> source) {     Objects.requireNonNull(source);     final Product product = source.getT1();     final BigInteger quantity = source.getT2();     final String userName = source.getT3();      return new UpdateOneModel&lt;>(       Filters.and(         Filters.eq(Product.SIZE_DB_FIELD, product.getSize().name()),         Filters.eq(Product.ARTICLE_DB_FIELD, product.getArticle())       ),       Document.parse(         String.format(           \"{ $inc: { %s: %d } }\",           Product.QUANTITY_DB_FIELD,           quantity         )       ).append(         \"$set\",         new Document(           Product.LAST_MODIFIED_BY_DB_FIELD,           userName         )       ),       new UpdateOptions().upsert(false)     );   } }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>Firstly, find a document by article and size. Figure 2 shows that we have a compound index on the size and article fields of the product collection to facilitate such a search;<\/li>\n<li>Increment the quantity of the found document and set the name of the user in the <code>lastModifiedBy<\/code> field;<\/li>\n<li>It is also possible to <code>upsert<\/code> a document here, but we are interested only in the modification of the existing documents in the storage.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Now we are ready to implement the central part of our processing which is the <code>incrementProductQuantity<\/code> method of the <code>ProductPatcherDaoImpl<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@Override public Mono&lt;BulkWriteResult> incrementProductQuantity(   final String fileName,   final List&lt;UpdateOneModel&lt;Document>> models,   final String userName ) {   return transactionalOperator.execute(     action -> reactiveMongoOperations.getCollection(Product.COLLECTION_NAME)       .flatMap(collection ->         Mono.from(collection.bulkWrite(models, new BulkWriteOptions().ordered(true)))        ).&lt;BulkWriteResult>handle((bulkWriteResult, synchronousSink) -> {         final int fileCount = models.size();         if (Objects.equals(bulkWriteResult.getModifiedCount(), fileCount)) {           synchronousSink.next(bulkWriteResult);         } else {           synchronousSink.error(             new IllegalStateException(               String.format(                 \"Inconsistency between modified doc count: %d and file doc count: %d. Please, check file: %s\",                 bulkWriteResult.getModifiedCount(), fileCount, fileName               )             )           );         }        }).onErrorResume(         e -> Mono.fromRunnable(action::setRollbackOnly)           .log(\"Exception while incrementProductQuantity: \" + fileName + \": \" + e)           .then(Mono.empty())       )   ).singleOrEmpty(); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>Use a <code>transactionalOperator<\/code> to roll back a transaction manually. As has been mentioned before, our goal is to process all files while skipping those causing exceptions;<\/li>\n<li>Run a single sub-process to bulk write modifications to the database sequentially for fail-fast and less resource-intensive behavior. The word &#171;single&#187; is of paramount importance here because we avoid the dangerous &#171;N+1 Query Problem&#187; leading to spawning a lot of sub-processes on a <code>flux<\/code> within <code>flatMap<\/code>;<\/li>\n<li><code>Handle<\/code> the situation when the number of the documents processed does not match the one coming from the distinct number of the products in the file;<\/li>\n<li>The <code>onErrorResume<\/code> method handles the rollback of the transaction and then returns <code>Mono.empty()<\/code> to skip the current processing;<\/li>\n<li>Expect either a single item or an empty Mono as the result of the <code>transactionalOperator.execute<\/code> method.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>One would say: &#171;You called <code>collection.bulkWrite(models, new BulkWriteOptions().ordered(true))<\/code>, what about setting a session?&#187;. The thing is that the <code>SessionAwareMethodInterceptor<\/code> of the Spring Data MongoDB does it via reflection:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">ReflectionUtils.invokeMethod(targetMethod.get(), target,         prependSessionToArguments(session, methodInvocation)<\/code><\/pre>\n<p>  <\/p>\n<p>Here is the <code>prependSessionToArguments<\/code> method:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private static Object[] prependSessionToArguments(ClientSession session, MethodInvocation invocation) {    Object[] args = new Object[invocation.getArguments().length + 1];    args[0] = session;   System.arraycopy(invocation.getArguments(), 0, args, 1, invocation.getArguments().length);    return args; }<\/code><\/pre>\n<p>  <\/p>\n<p>1) Get the arguments of the <code>MethodInvocation<\/code>;<br \/>  2) Add <code>session<\/code> as a the first element in the <code>args<\/code> array.<\/p>\n<p>  <\/p>\n<p>In fact, the following method of the <code>MongoCollectionImpl<\/code> is called:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@Override public Publisher&lt;BulkWriteResult> bulkWrite(final ClientSession clientSession,                                             final List&lt;? extends WriteModel&lt;? extends TDocument>> requests,                                             final BulkWriteOptions options) {   return Publishers.publish(     callback -> wrapped.bulkWrite(clientSession.getWrapped(), requests, options, callback)); }<\/code><\/pre>\n<p>  <\/p>\n<h3 id=\"3-test-implementation\">3. Test implementation<\/h3>\n<p>  <\/p>\n<p>So far so good, we can create integration tests to cover our logic.<\/p>\n<p>  <\/p>\n<p>To begin with, we create <code>ProductControllerITTest<\/code> to test our public API via the Spring\u2019s <code>WebTestClient<\/code> and initialize a MongoDB instance to run tests against:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">private static final MongoDBContainer MONGO_DB_CONTAINER =   new MongoDBContainer(\"mongo:4.2.8\");<\/code><\/pre>\n<p>  <\/p>\n<p>1) Use a static field to have single Testcontainers\u2019 <code>MongoDBContainer<\/code> per all test methods in <code>ProductControllerITTest<\/code>;<br \/>  2) We use 4.2.8 MongoDB container version from Docker Hub as it is the latest stable one, otherwise <code>MongoDBContainer<\/code> defaults to 4.0.10.<\/p>\n<p>  <\/p>\n<p>Then in static methods <code>setUpAll<\/code> and <code>tearDownAll<\/code> we start and stop the <code>MongoDBContainer<\/code> respectively. Even though we do not use Testcontainers&#8217; reusable feature here, we leave open the possibility of setting it. Which is why we call <code>MONGO_DB_CONTAINER.stop()<\/code> only if the reusable feature is turned off.<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@BeforeAll static void setUpAll() {     MONGO_DB_CONTAINER.start(); }  @AfterAll static void tearDownAll() {   if (!MONGO_DB_CONTAINER.isShouldBeReused()) {     MONGO_DB_CONTAINER.stop();   } }<\/code><\/pre>\n<p>  <\/p>\n<p>Next we set <code>spring.data.mongodb.uri<\/code> by executing <code>MONGO_DB_CONTAINER.getReplicaSetUrl()<\/code> in <code>ApplicationContextInitializer<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">static class Initializer implements ApplicationContextInitializer&lt;ConfigurableApplicationContext> {   @Override   public void initialize(@NotNull ConfigurableApplicationContext configurableApplicationContext) {     TestPropertyValues.of(       String.format(\"spring.data.mongodb.uri: %s\", MONGO_DB_CONTAINER.getReplicaSetUrl())     ).applyTo(configurableApplicationContext);   } }<\/code><\/pre>\n<p>  <\/p>\n<p>Now we are ready to write a first test without any transaction collision, because our test files (see Figure 3) have products whose articles do not clash with one another.<\/p>\n<p>  <\/p>\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/ed7\/310\/0ec\/ed73100ec821f604f43a623162e49623.png\" alt=\"Figure 3\" title=\"Excel files causing no collision in the articles of the products\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/ed7\/310\/0ec\/ed73100ec821f604f43a623162e49623.png\"\/><br \/>  <em>Figure 3 Excel files causing no collision in the articles of the products<\/em><\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@WithMockUser(   username = SecurityConfig.ADMIN_NAME,   password = SecurityConfig.ADMIN_PAS,   authorities = SecurityConfig.WRITE_PRIVILEGE ) @Test void shouldPatchProductQuantity() {   \/\/GIVEN   insertMockProductsIntoDb(Flux.just(product1, product2, product3));   final BigInteger expected1 = BigInteger.valueOf(16);   final BigInteger expected2 = BigInteger.valueOf(27);   final BigInteger expected3 = BigInteger.valueOf(88);   final String fileName1 = \"products1.xlsx\";   final String fileName3 = \"products3.xlsx\";   final String[] fileNames = {fileName1, fileName3};   final FileUploadDto fileUploadDto1 = ProductTestUtil.mockFileUploadDto(fileName1, 2);   final FileUploadDto fileUploadDto3 = ProductTestUtil.mockFileUploadDto(fileName3, 1);    \/\/WHEN   final WebTestClient.ResponseSpec exchange = webClient     .patch()     .uri(BASE_URL)     .contentType(MediaType.MULTIPART_FORM_DATA)     .body(BodyInserters.fromMultipartData(ProductTestUtil.getMultiPartFormData(fileNames)))     .exchange();    \/\/THEN   exchange.expectStatus().isAccepted();    exchange.expectBodyList(FileUploadDto.class)     .hasSize(2)     .contains(fileUploadDto1, fileUploadDto3);    StepVerifier.create(productDao.findAllByOrderByQuantityAsc())     .assertNext(product -> assertEquals(expected1, product.getQuantity()))     .assertNext(product -> assertEquals(expected2, product.getQuantity()))     .assertNext(product -> assertEquals(expected3, product.getQuantity()))     .verifyComplete(); }<\/code><\/pre>\n<p>  <\/p>\n<p>Finally, let us test a transaction collision in action, keeping in mind Figure 1 and Figure 4 showing such files:<\/p>\n<p>  <\/p>\n<p><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/7bc\/b7c\/94c\/7bcb7c94c7b965f4f7803c811aff1e9e.png\" alt=\"Figure 4\" title=\"Excel files causing a collision in the articles of the products\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/7bc\/b7c\/94c\/7bcb7c94c7b965f4f7803c811aff1e9e.png\"\/><br \/>  <em>Figure 4 Excel files causing a collision in the articles of the products<\/em><\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@WithMockUser(   username = SecurityConfig.ADMIN_NAME,   password = SecurityConfig.ADMIN_PAS,   authorities = SecurityConfig.WRITE_PRIVILEGE ) @Test void shouldPatchProductQuantityConcurrently() {   \/\/GIVEN   TransactionUtil.setMaxTransactionLockRequestTimeoutMillis(     20,     MONGO_DB_CONTAINER.getReplicaSetUrl()   );   insertMockProductsIntoDb(Flux.just(product1, product2));   final String fileName1 = \"products1.xlsx\";   final String fileName2 = \"products2.xlsx\";   final String[] fileNames = {fileName1, fileName2};   final BigInteger expected120589Sum = BigInteger.valueOf(19);   final BigInteger expected120590Sum = BigInteger.valueOf(32);   final BigInteger expected120589T1 = BigInteger.valueOf(16);   final BigInteger expected120589T2 = BigInteger.valueOf(12);   final BigInteger expected120590T1 = BigInteger.valueOf(27);   final BigInteger expected120590T2 = BigInteger.valueOf(11);   final FileUploadDto fileUploadDto1 = ProductTestUtil.mockFileUploadDto(fileName1, 2);   final FileUploadDto fileUploadDto2 = ProductTestUtil.mockFileUploadDto(fileName2, 2);    \/\/WHEN   final WebTestClient.ResponseSpec exchange = webClient     .patch()     .uri(BASE_URL)     .contentType(MediaType.MULTIPART_FORM_DATA)     .accept(MediaType.APPLICATION_STREAM_JSON)     .body(BodyInserters.fromMultipartData(ProductTestUtil.getMultiPartFormData(fileNames)))     .exchange();    \/\/THEN   exchange.expectStatus().isAccepted();   assertThat(     extractBodyArray(exchange),     either(arrayContaining(fileUploadDto1))       .or(arrayContaining(fileUploadDto2))       .or(arrayContainingInAnyOrder(fileUploadDto1, fileUploadDto2))   );    final List&lt;Product> list = productDao.findAll(Sort.by(Sort.Direction.ASC, \"article\"))     .toStream().collect(Collectors.toList());   assertThat(list.size(), is(2));    assertThat(     list.stream().map(Product::getQuantity).toArray(BigInteger[]::new),     either(arrayContaining(expected120589T1, expected120590T1))       .or(arrayContaining(expected120589T2, expected120590T2))       .or(arrayContaining(expected120589Sum, expected120590Sum))   );   TransactionUtil.setMaxTransactionLockRequestTimeoutMillis(     5,     MONGO_DB_CONTAINER.getReplicaSetUrl()   ); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>We can specify the maximum amount of time in milliseconds that multi-document transactions should wait to acquire locks required by the operations in the transaction (by default, multi-document transactions wait 5 milliseconds);<\/li>\n<li>As an example here, we might use a helper method to change 5ms with 20ms (see an implementation details below).<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Note that the <code>maxTransactionLockRequestTimeoutMillis<\/code> setting has no sense for this particular test case and serves the purpose of the example. After running this test class 120 times via a script <code>.\/load_test.sh 120 ProductControllerITTest.shouldPatchProductQuantityConcurrently<\/code> in the tools directory of the project, I got the following figures:<\/p>\n<p>  <\/p>\n<div class=\"scrollable-table\">\n<table>\n<thead>\n<tr>\n<th>indicator<\/th>\n<th>20ms,<br \/>  times<\/th>\n<th>5ms(default),<br \/>  times<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>T1 successes<\/td>\n<td>61<\/td>\n<td>56<\/td>\n<\/tr>\n<tr>\n<td>T2 successes<\/td>\n<td>57<\/td>\n<td>63<\/td>\n<\/tr>\n<tr>\n<td>T1 and T2 success<\/td>\n<td>2<\/td>\n<td>1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>  <\/p>\n<p><em>Figure 5 Running the shouldPatchProductQuantityConcurrently test 120 times with 20 and 5 ms maxTransactionLockRequestTimeoutMillis respectively<\/em><\/p>\n<p>  <\/p>\n<p>While going through logs, we may come across something like:<\/p>\n<p>  <\/p>\n<blockquote><p>Exception while incrementProductQuantity: products1.xlsx: com.mongodb.MongoCommandException: Command failed with error 112 (WriteConflict): &#8216;WriteConflict&#8217; on server\u2026<br \/>  Initiating transaction rollback\u2026<br \/>  Initiating transaction commit\u2026<br \/>  About to abort transaction for session\u2026<br \/>  About to commit transaction for session&#8230;<\/p><\/blockquote>\n<p>Then, let us test the processing of the big file containing 1 million products in a separate <code>PatchProductLoadITTest<\/code>:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">@WithMockUser(   username = SecurityConfig.ADMIN_NAME,   password = SecurityConfig.ADMIN_PAS,   authorities = SecurityConfig.WRITE_PRIVILEGE ) @Test void shouldPatchProductQuantityBigFile() {   \/\/GIVEN   unzipClassPathFile(\"products_1M.zip\");    final String fileName = \"products_1M.xlsx\";   final int count = 1000000;   final long totalQuantity = 500472368779L;   final List&lt;Document> products = getDocuments(count);    TransactionUtil.setTransactionLifetimeLimitSeconds(     900,     MONGO_DB_CONTAINER.getReplicaSetUrl()   );    StepVerifier.create(     reactiveMongoTemplate.remove(new Query(), Product.COLLECTION_NAME)       .then(reactiveMongoTemplate.getCollection(Product.COLLECTION_NAME))       .flatMapMany(c -> c.insertMany(products))       .switchIfEmpty(Mono.error(new RuntimeException(\"Cannot insertMany\")))       .then(getTotalQuantity())   ).assertNext(t -> assertEquals(totalQuantity, t)).verifyComplete();    \/\/WHEN   final Instant start = Instant.now();   final WebTestClient.ResponseSpec exchange = webClient     .patch()     .uri(BASE_URL)     .contentType(MediaType.MULTIPART_FORM_DATA)     .accept(MediaType.APPLICATION_STREAM_JSON)     .body(BodyInserters.fromMultipartData(ProductTestUtil.getMultiPartFormData(\"products_1M.xlsx\")))     .exchange();    \/\/THEN   exchange     .expectStatus()     .isAccepted()     .expectBodyList(FileUploadDto.class)     .contains(ProductTestUtil.mockFileUploadDto(fileName, count));   StepVerifier.create(getTotalQuantity())     .assertNext(t -> assertEquals(totalQuantity * 2, t))     .verifyComplete();   log.debug(     \"============= shouldPatchProductQuantityBigFile elapsed {}minutes =============\",     Duration.between(start, Instant.now()).toMinutes()   ); }<\/code><\/pre>\n<p>  <\/p>\n<ol>\n<li>The general setup is similar to the <code>ProductControllerITTest<\/code>;<\/li>\n<li>Unzip a json file containing 1 million products which requires about 254M on a disk;<\/li>\n<li>Transactions have a lifetime limit as specified by <code>transactionLifetimeLimitSeconds<\/code> which is 60 seconds by default. We need to increase it here, because generally it takes more than 60 s to process such a file. For this, we use a helper method to change this lifespan to 900 s (see the implementation details below). For your information, the REST call with the file takes GitHub Actions about 9-12 minutes;<\/li>\n<li>Before processing, we clean up a product collection, insert 1 million products from the json file and then get the total of the quantity;<\/li>\n<li>Given the products in the json file and the big excel file are equal, we assert that the total quantity of the product after processing should double.<\/li>\n<\/ol>\n<p>  <\/p>\n<p>Such a test requires a relatively big heap of about 4GB (see Figure 6) and Docker&#8217;s memory resource of about 6GB (see Figure 7):<br \/>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/351\/cea\/1e7\/351cea1e7f1614d0bd1ca8025774b570.png\" alt=\"Figure 6\" title=\"VisualVM Monitor Heap while uploading a 1-million-product file\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/351\/cea\/1e7\/351cea1e7f1614d0bd1ca8025774b570.png\"\/><br \/>  <em>Figure 6 VisualVM Monitor Heap while uploading a 1-million-product file<\/em><br \/>  <img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/post_images\/923\/586\/885\/923586885f7301f86c2fb919362cd542.png\" alt=\"Figure 7\" title=\"Cadvisor memory total usage while uploading a 1-million-product file\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/post_images\/923\/586\/885\/923586885f7301f86c2fb919362cd542.png\"\/><br \/>  <em>Figure 7 Cadvisor memory total usage while uploading a 1-million-product file<\/em><\/p>\n<p>  <\/p>\n<p>As we can see, it is sensible to configure the maximum amount of disk space allowed for file parts and the maximum number of parts allowed in a given multipart request. Which is why I added properties to a proper application.yml file and then set them in the <code>configureHttpMessageCodecs<\/code> method of the implemented <code>WebFluxConfigurer<\/code>. However, adding Rate Limiter and configuring <code>Schedulers<\/code> might be a better solution in production environment. Note that we use <code>Schedulers.boundedElastic()<\/code> here having a pool of <code>10 * Runtime.getRuntime().availableProcessors()<\/code> threads by default.<\/p>\n<p>  <\/p>\n<p>Here is <code>TransactionUtil<\/code>containing the above-mentioned helper methods:<\/p>\n<p>  <\/p>\n<pre><code class=\"java\">public class TransactionUtil {   private TransactionUtil() {   }    public static void setTransactionLifetimeLimitSeconds(     final int duration,     final String replicaSetUrl   ) {     setMongoParameter(\"transactionLifetimeLimitSeconds\", duration, replicaSetUrl);   }    public static void setMaxTransactionLockRequestTimeoutMillis(     final int duration,     final String replicaSetUrl   ) {     setMongoParameter(\"maxTransactionLockRequestTimeoutMillis\", duration, replicaSetUrl);   }    private static void setMongoParameter(     final String param,     final int duration,     final String replicaSetUrl   ) {     try (final MongoClient mongoReactiveClient = MongoClients.create(       ConnectionUtil.getMongoClientSettingsWithTimeout(replicaSetUrl)     )) {        StepVerifier.create(mongoReactiveClient.getDatabase(\"admin\").runCommand(         new Document(\"setParameter\", 1).append(param, duration)       )).expectNextCount(1)         .verifyComplete();     }   } }<\/code><\/pre>\n<p>  <\/p>\n<h3 id=\"4-how-can-i-play-with-the-code\">4. How can I play with the code?<\/h3>\n<p>  <\/p>\n<p><a href=\"https:\/\/github.com\/silaev\/wms\" rel=\"nofollow\">Small WMS (warehouse management system) on GitHub<\/a>.<\/p>\n<p>  <\/p>\n<h3 id=\"5-whats-in-it-for-me\">5. What\u2019s in it for me?<\/h3>\n<p>  <\/p>\n<ol>\n<li>The <code>MongoDBContainer<\/code> takes care of the complexity in the MongoDB replica set initialization allowing the developer to focus on testing. Now we can simply make MongoDB transaction testing part of our CI\/CD process; <\/li>\n<li>While processing data, it is sensible to favor MongoDB\u2019s bulk methods, reducing the number of sub-processes within the <code>flatMap<\/code> method of the <code>Flux<\/code> and thus to avoid introducing the &#171;N+1 Query problem&#187;. However, it also comes at a price because here we need to collect a list of <code>UpdateOneModel<\/code> and keep it in memory lacking reactive flexibility;<\/li>\n<li>When it comes to skipping processing, one might employ <code>onErrorResume<\/code> instead of <a href=\"https:\/\/github.com\/reactor\/reactor-core\/issues\/2184\" rel=\"nofollow\"><code>the dangerous onErrorContinue<\/code><\/a><\/li>\n<li>Even though are we allowed to set <code>maxTransactionLockRequestTimeoutMillis<\/code> and <code>transactionLifetimeLimitSeconds<\/code> as parameters during start-up to mongod, we may achieve the effect by calling the MongoDB&#8217;s <code>adminCommand<\/code> via helper methods;<\/li>\n<li>Processing big files is resource-consuming and thus better be limited.<\/li>\n<\/ol>\n<p>  <\/p>\n<h3 id=\"6-want-to-go-deeper\">6. Want to go deeper?<\/h3>\n<p>  <\/p>\n<p>To construct a multi-node MongoDB replica set for testing complicated failover cases, consider <a href=\"https:\/\/github.com\/silaev\/mongodb-replica-set\" rel=\"nofollow\">the mongodb-replica-set project<\/a>.<\/p>\n<p>  <\/p>\n<h3 id=\"7-links\">7. Links<\/h3>\n<p>  <\/p>\n<ol>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=8TkY_RaoLCQ\" rel=\"nofollow\">Reactive Transactions Masterclass by Michael Simons &amp; Mark Paluch<\/a><\/li>\n<li><a href=\"https:\/\/docs.spring.io\/spring-data\/mongodb\/docs\/current\/reference\/html\/#reference\" rel=\"nofollow\">Spring Data MongoDB \u2014 Reference Documentation<\/a><\/li>\n<li><a href=\"https:\/\/docs.mongodb.com\/manual\/core\/transactions\/\" rel=\"nofollow\">MongoDB Transactions<\/a><\/li>\n<li><a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/js-collection\/\" rel=\"nofollow\">MongoDB Collection Methods<\/a><\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/513026\/\"> https:\/\/habr.com\/ru\/articles\/513026\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-1\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h3 id=\"1-introduction\">1. Introduction<\/h3>\n<p>  <\/p>\n<p>How can I easily test my MongoDB multi-document transaction code without setting up MongoDB on my device? One might argue that they have to set it up first because in order to carry out such a transaction it needs a session which requires a replica set. Thankfully, there is no need to create a 3-node replica set and we can run these transactions only against a single database instance.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-405683","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/405683","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=405683"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/405683\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=405683"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=405683"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=405683"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}