The Comprehensive Architectural Guide: Building Full-Text Search in MongoDB Atlas Without Elasticsearch

There are few impediments in today’s software architectures as stubborn as the conflict between fast transactional writes and flexible, comprehensive search capabilities. Databases were always designed to excel at one of these capabilities or the other. Both relational and document databases like MongoDB were built to provide operational stability, excellent consistency, and high-throughput transactional operations for CRUD tasks. On the other hand, specialized search engines functioning on the principle of unstructured text processing, relevance determination, fuzzy matching, and real-time suggestion were being established like Elasticsearch.

In order to help to overcome this problem, engineering teams turned to the dual-clustering system. In this scheme, one data cluster serves as the main database entity with another search node running parallel to it. In order to synchronize two clusters, developers implement different synchronization pipelines using change data capture (CDC) tools, message queues like Apache Kafka, or custom-built ETL scripts created with the help of Logstash.

Although the dual-cluster approach addressed short-term functional requirements, it created a lot of architectural issues. The use of separate infrastructure systems led to increased maintenance costs related to cluster management and higher cloud hosting costs due to redundancy in architecture. Moreover, the necessity to synchronize two isolated systems means dealing with replication delays, consistency problems, and data drift. If something goes wrong while synchronizing data, users might encounter missing search results which would result in spending time on investigating how the system synchronizes data through pipelines.

Another solution is the implementation of MongoDB Atlas Search that integrates the industry-standard Apache Lucene search engine into the main database cluster, making it possible to avoid additional search platforms and sync pipelines. As a result, in MongoDB Atlas, users can access and modify the needed data through Lucene indexes without waiting for the completion of the synchronization process.

Atlas Search brings together transactional data storage, full-text search and vector search on a single cloud platform and, as such, eliminates the need for any external search clusters. In case you are upskilling yourself through a thorough MongoDB online course or working on changing your enterprise architecture, this guide shows you how to tear down the old two-cluster pipeline, create static search indexes, and build query pipelines.

The Nightmare of Dual-Cluster Operations

For the transformational potential of integrated search to be fully established, we will need to first analyze the shortcomings of the dual database environment.

In the classic dual-cluster setup, each and every document that gets written to MongoDB has to be replicated into Elasticsearch. Several architectural risks are introduced by this solution:

1. Eventual Consistency and Replication Lag MongoDB changes are not immediately propagated to Elasticsearch. Replication lag may lead to incorrect or outdated search results, for example in the presence of peaks of write operations, emergence of unreachable network nodes, or a client crash.

2. Overhead of Double Storage: Storing full-text string fields in two separate cloud deployments leads to a doubling of the infrastructure and the costs associated with receiving data through the network.

3. Data Drift and Unsynchronization: In the case when the data transfer fails in Elasticsearch or in the ETL pipeline, the search cluster may fall behind the main source of data, thus making re-indexing necessary.

4. Complexity for Development and Client Overhead: Software engineers have to work with two SDKs (mongodb and @elastic/elasticsearch). Getting search results employs a convoluted scheme of operations: first, it is necessary to search through Elasticsearch for document IDs, and only after that it is possible to send a query to MongoDB.

   

The Architecture of Atlas Search: Under the Hood

MongoDB Atlas Search has intelligent framework engineer to bypass resource contention between database read/write operations and search processing.

The Atlas server instance runs two different daemon processes:

• mongod : The real database engine process. It deals with the oplogs, B-Tree indexing, WiredTiger caching, and CRUD API operations.

• mongot: Java process with Apache Lucene inside. This process takes care of the indexing, tokenization, stemming, building the inverted index and processing the search queries.

How Synchronized Data Transfer Works

If a write operation is performed in mongod, it gets committed to the local transaction log (oplog). Change streams keep streaming this operation into mongot, which updates an in-memory Lucene inverted index and saves it back to disk. It happens in sub-milliseconds without any ETL abstractions exposed to the developer.

Isolated Workloads: Dedicated Search Nodes

In the case of a large-scale production environment where mongod and mongot run on the same server, there might be CPU/RAM bottleneck while background indexing is going on.

Atlas provides the option of Dedicated Search Nodes. Such architecture provides physical separation: mongod is hosted on database nodes and mongot is hosted on isolated compute instances optimized for inverted index memory needs. Your search engine compute is now independent of your database cluster compute.

Creating Index Mappings for Advanced Search

In contrast to the standard MongoDB B-Tree indexes generated using the createIndex() method, Atlas Search indexes are inverted indexes created with the help of Lucene.

It is possible to set indexes in two ways: Dynamic Mapping and Static Mapping.

• Dynamic Mapping (“dynamic”: true): Automatically indexes data in any fields (strings, numbers, booleans) of any document. It should be used for rapid development or where the schema cannot be predicted.

• Static Mapping (“dynamic”: false): Indicates the paths that will be indexed and the special analyzers for the fields specified. Recommended for production use since the RAM resources will be used more efficiently.

Definition of Enterprise Static Mapping

Here’s a static index mapping for an enterprise content platform that stores articles, metadata and tags:

Primary Field Caps

1. Analyzers: lucene.english does tokenization; gets rid of common stop words (for example, the, is, at), converts characters to lower case, and applies algorithmic stemming (that is, it reduces such words as running, runs, and ran to the base term run).

2. Multi-Type Indicator: The title field utilizes multi-index. The main definition allows one to make full-text search, while keyword sub-field applies lucene.keyword to keep a string unchanged verbatim for sorting and exact phrases.

3. Autocomplete Tokenizing: The edgeGram tokenizing technique generates n-grams from the beginning of the word making it possible to use search-as-you-type.

4. Synonym Indicating: The synonyms mapping refers to the external collection where mapped phrases are stored.

Searching for Atlas via Aggregation Pipeline

In the case of MongoDB Atlas, search operations are performed using the regular aggregation pipeline stages by means of $search or $searchMeta stages. It is common for practical mongodb training online sessions to be based on learning these pipeline stages since one has to go beyond basic CRUD operations.

Pattern 1: Relevance ranked fuzzy text search

Detecting typos and misspellings is important for search user interfaces. Atlas Search uses a Damerau-Levenshtein algorithm to compute string edit distances.

Pattern 2: Search-as-You-Type (Auto-complete)

Our static mapping, defined by the edgeGram analyzer, allows the building of predictive search interfaces:

Pattern 3: Compound Queries with Complex Conditions (must, mustNot, should, filter)

This compound operator enables developers to build complex search queries with logical AND/OR/NOT conditions, similar to the boolean query compositions of Elasticsearch.

Advanced Search Mechanics

Faceted Search ($searchMeta)

Countless filters can be applied to the facet search results, allowing customers to immediately find what they’re looking for in terms of category, price range, etc.

Faceted search can be executed with the help of $searchMeta where aggregate values will be returned instead of actual documents.

Cross-Document Joins ($lookup) & In-Engine Filtering

One of the most attractive features of MongoDB Atlas Search is its ability to execute text queries, conduct relational joins, and perform standard pipeline filters within a single atomic database request.

Moving to a New Level of Search by Using Vectors and Hybrid Methods

Modern applications are demanding semantic search which allows the ability to find documents according to their context, not just exact matching of keywords.

The new MongoDB Atlas offers the new capability of Atlas Vector Search natively as part of full-text search. That allows you to keep the vector embedding of your documents directly created by models such as OpenAI, Cohere, or Hugging Face within documents.

Using Hybrid Search with Reciprocal Rank Fusion

With the help of MongoDB's $rankFusion operator, the combination of lexical $search and semantic vector search $vectorSearch can be done automatically providing you with the results of the search that are both precise according to keywords and have relevance based on the meanings.

Performance Tuning, Sizing, and Best Practices

Follow these sizing guidelines for high availability and low latency under production load:

1. Node size in search

Calculate the total disk footprint of your search indices. The whole Lucene inverted index must reside in the RAM cache of the OS to get the best performance with Lucene and to avoid disk page faults.

• Rule of Thumb: Use Dedicated Search Total memory of nodes $\geq 1.2 \times$ Total Inverted Index Size

2. Considerations for Oplog Sizing

mongot listens for updates on internal change streams, so if mongod is seeing high write-throughput bursts, and the replication oplog is undersized, the search index sync can fall behind.

• For heavy bulk-insert operations, always increase the minimum oplog size of the cluster to avoid mongot falling off the oplog.

3. Pipeline Efficiency Regulations

• $search First, $search MUST be stage 1.

• Push Filters into $search: Do not write typical $match stages immediately after $search if those fields can be indexed within $search.compound.filter. Lucene filtering in the engine is many orders of magnitude faster than evaluating database documents downstream in mongod.

• Early Use Projection: Following $search, eliminate unnecessary large document payload fields with a $project stage to reduce inter-process network overhead between mongot and mongod.

Modern Artificial Intelligence Tasks: Hybrid Search with Atlas Vector Search

More and more applications have the need for semantic search, which is to make queries based on meaning rather than directly matching keywords. MongoDB Atlas has native support for Vector Search in conjunction with full-text search.

Hybrid Search with Reciprocal Rank Fusion

Combining the conventional keyword phonetic search ($search) and semantic vector-based search ($vectorSearch) takes the applications' precision to a whole new level.

The $rankFusion feature by MongoDB relies on the integration of both types of search technology.

Implications of Architectural Decisions

Though MongoDB Atlas Search is over 90% capable for all search requirements across web-based, mobile, SaaS, and e-commerce applications, finalizing on the right architecture boils down to analyzing the needs of the specific application versus operational trade-offs involved.

1. Use through application in e-commerce catalogs, content portals, & SaaS: MongoDB Atlas Search is the best-suited option since its integration with operational data does away with synchronization delays, allows filtering over multiple fields using the aggregation framework effortlessly, and also helps save costs on cloud hosting by eliminating additional database clusters.

2. Use through application in search-as-you-type and auto-completion: MongoDB Atlas Search is the best option supplemented with the edgeGram tokenization which uses prefix queries in sub milliseconds through standard database drivers, thus eliminating the need of any external cache to be implemented.

3. Use through application in modern hybrid searches (i.e., Lexical + AI vector): MongoDB Atlas Search is the top choice in this type of searches as employing their native $rankFusion pipeline stage just permits developers to harmonically combine traditional scoring of keywords with AI vector embeddings ($vectorSearch) in single query without need to use any special vector database.

4. Significant unstructured log analytics: Dedicated Elasticsearch (ELK) cluster has been the industry benchmark. If your target for log analysis is processing terabytes of unprocessed log data or utilizing Kibana dashboards extensively for live insight purposes, the special ELK ecosystem cannot be surpassable.

5. Special low-level Lucene modifications: Dedicated Elasticsearch cluster is still a must if your system requires utilizing custom Java plugins or proprietary tokenizers built directly into the underlying Lucene core, as Atlas Search will manage the internal engine ecosystem on your behalf.

Conclusion: Fewer Clusters, Faster Delivery

Historically, traditional software architecture performed a clear-cut separation of tasks: transactional databases are responsible for storing the real-state information of any application, whereas separate external search clusters search for text and perform fuzzy matching. However, since new generations of applications rely on providing real-time and context-aware user experiences, ensuring the existence of two separate systems has created considerable technical debt and made systems less reliable.

Moving text-search operations from Elasticsearch to MongoDB Atlas Search radically changes the mentioned situation. With Atlas Search allowing for native integration of the well-known search engine Apache Lucene into other database applications, there is no need for the standardized ETL pipelines that are used to synchronize data. A complete absence of the external synchronicity layer also eliminates the problems of data drift, out-of-date search inquiries, and replication delay. Creating a new version of an official document means that the Lucene inverted indexing system will instantly rework the underlying indexing of the document without any special changes-making codes, message queues, or additional re-indexing processes.

Taking an operational approach to analyzing the situation, it can be said that bringing all these levels under one cloud platform results in substantial improvements in efficiency. There is no need for engineering teams to go through a long learning process in different stacks; rather, it only requires some investment in mongodb online training. Aggregation Pipeline stage instead of performing several queries, searching for IDs in the external cluster and later retrieving payloads from the database. It will make processes of developing, testing, and executing complex queries faster and more efficient.

Thus, besides its operational simplicity, the adoption of Atlas Search solves financial and technical scaling issues. When text fields are stored once but not duplicated in separate cloud systems, the costs of storage dramatically decrease, and the expenses of egressing data between clusters diminish completely. In case there is a need to scale computation due to high volume of requests, Atlas Search provides its users with the possibility to support dedicated search nodes, which allow system designers to make the RAM and CPU of the search-indexing independent of other database nodes.

Additionally, unification ensures that applications are ready to perform modern machine learning-based workloads. This means that, since Atlas is capable of processing vector embeddings together with Lucene indexing, there is no need for engineers to use a separate vector search engine for implementing hybrid search by using reciprocal rank fusion ($rankFusion).

Finally, switching from an external Elasticsearch cluster to MongoDB Atlas Search is not only a means of reducing costs but also a structural decision. As a result of removing synchronization issues, making backend easier, and building a unified data platform for CRUD, full-text search, and vector embeddings, companies can move features faster, make systems more reliable, and dedicate people to business innovations instead of infrastructure.