Corpus Analysis Techniques

Want to compare texts for meaning (not just words) but prefer to keep things local? Try running a MiniLM model on your laptop using the sentence-transformers module. ✨ Language Models - simplified: This MiniLM model "all-MiniLM-L6-v2" has been trained to pair up sentences based on their meaning, creating a pool  where "similar things" are grouped close together (a dense vector space of 384 dimensions). When you provide texts, it maps each one into this space. You can measure the angle between their vectors to see how similar they are. The closer the angle is to 0, the more alike the sentences are. This is termed cosine similarity. ⏩ Steps: 1. Import the sentence-transformers module and load the model. 2. Generate vectors for each text. 3. Calculate cosine similarity and sort pairs by similarity (descending order). 4. Review the output ✅ The result: Instead of manually reviewing all pairs, we can focus our efforts on those sitting in the middle of the scale. Pairs at the extremes (values close to +1 or -1) are likely to be accurately judged by the model already. This approach cuts down the time and effort needed. 🚀 🏆 Use case: I found this technique while trying to validate item descriptions in bulk for a data pipeline I’m building at work. With thousands of lines to review, manually eyeballing them wasn’t just exhausting — it was highly error-prone. Using this method locally keeps the data confidential while producing faster and more accurate results. That said, the output still needs human review — leaving it entirely to a machine would be irresponsible. This is a great example of how we can leverage machine learning to support us rather than replace us. While I await approval for my PoC, feel free to share any text analysis techniques you’ve found useful, I’d love to hear them! #SemanticTextualSimilarity #DataScience #ML #MachineLearning #DataEngineering #DataPipeline #HuggingFace #SentenceTransformers #LLM #LargeLanguageModel #MiniLM

Understanding Bag of Words (BoW): A Simple Explanation with Examples The Bag of Words (BoW) model is a foundational concept in Natural Language Processing (NLP). It is widely used to convert text into numerical representations for machine learning tasks. This post provides a clear explanation of BoW, its implementation, and an example to clarify its workings. What is Bag of Words? The Bag of Words model simplifies text data by: 1)Treating a document as a "bag" of its words. 2)Ignoring grammar, word order, and context. 3)Focusing solely on the presence or frequency of words in a document. This simplicity makes it a great starting point for many NLP applications, such as text classification, information retrieval, and topic modeling. How Bag of Words Works The BoW process involves a few straightforward steps: 1. Tokenization Break the text into individual words (tokens). Example: Document 1: "The dog barked at the cat." Tokens: ['The', 'dog', 'barked', 'at', 'the', 'cat'] 2. Normalization (Optional) Convert all words to lowercase to avoid case sensitivity. Remove punctuation and stop words if needed (e.g., words like "the," "on," and "at"). 3. Build a Vocabulary Collect all unique words from the entire corpus (collection of documents). Example Vocabulary: ['the', 'dog', 'barked', 'at', 'cat', 'sat', 'on', 'mat'] 4. Create Vectors For each document, create a vector with the same length as the vocabulary. Populate each vector with word frequencies from the document. Example: Bag of Words in Action Let’s take a simple corpus with two documents: Document 1: "The dog barked at the cat." Document 2: "The cat sat on the mat." Step 1: Vocabulary From the corpus, the unique words (vocabulary) are: ['the', 'dog', 'barked', 'at', 'cat', 'sat', 'on', 'mat'] Step 2: Frequency Count Count how often each word in the vocabulary appears in each document Step 3: Create Vectors Using the frequency counts, each document is represented as a vector: Document 1: [2, 1, 1, 1, 1, 0, 0, 0] Document 2: [2, 0, 0, 0, 1, 1, 1, 1] Why is "2" the Value for "the"? The word "the" appears twice in each document, as it is used both at the beginning of the sentence and before certain nouns (e.g., "the cat," "the mat"). Hence, the frequency count for "the" is 2 in both vectors. Advantages of Bag of Words 1)Simple and Intuitive: Easy to understand and implement. 2)Effective for Small Datasets: Works well when the vocabulary is limited. 3)Baseline for NLP Tasks: Useful as a first step in many text processing workflows. Disadvantages of Bag of Words 1)Ignores Context and Meaning: The order and relationships between words are lost. 2)High Dimensionality: Large vocabularies lead to high-dimensional vectors. 3)Sparse Data: Most entries in the vector are zeros for large corpora.

The Evolution of Text Representation: From Bags of Words to Transformers 1) Bag of Words (BoW): The most basic approach, BoW simply counts how often each word appears in a document. While efficient, it ignores the order and relationships between words, leading to not understanding context. 2) TF-IDF (Term Frequency-Inverse Document Frequency): Building upon BoW, TF-IDF refines word importance by considering both frequency within the document and rarity across the corpus. Frequent words like "the" have low TF-IDF, while unique and relevant ones receive higher scores. 3) Word2Vec: This neural network model takes a leap by representing words as vectors in a high-dimensional space. Words with similar meanings have vectors closer together, capturing semantic relationships beyond simple word counts. 4) Recurrent Neural Networks (RNNs): Recognizing the sequential nature of text, RNNs use past outputs to make future predictions. LSTMs, a type of RNN, excel at capturing long-range dependencies, further enhancing context comprehension. 5) Transformers: It is a encoder-decoder model leveraging attention mechanisms that allows it to learn long-range dependencies between different parts of an input sequence. Attention allows a model to focus on specific parts of its input data. This enables learning complex relationships between words, regardless of their distance, leading to superior performance in various NLP tasks. #naturallanguageprocessing #machinelearning

Traditional ML methods for extracting topics from a corpus of messages rely on converting each message into a vector in some vector space and then clustering in that vector space. "Topics" are then just regions in that vector space. Even interpreting such regions is not trivial; editing them after the fit is almost impossible. Here we show a different way, using LLM calls only. The biggest advantage of this is that the topic descriptions _are_ the topics, so you can start with an initial set of human-defined topics and let the algorithm add others if required - and edit the latter if you wish. It works as follows: We feed one message at a time to the topic processor; it either assigns it to one of the existing topics or, if none are a good fit, puts it aside. Once the number of messages put aside reaches a threshold, these are used to extract a new topic, which is added to the list. There is also the option of generating topic hierarchies by setting `max_depth` to a value bigger than 1. Check out the example notebook at wise-topic: https://lnkd.in/dJMytKcm

Key topics to learn in GenAI - 𝗙𝗿𝗼𝗺 𝗪𝗼𝗿𝗱 𝗖𝗼𝘂𝗻𝘁𝘀 𝘁𝗼 𝗪𝗼𝗿𝗱 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝗰𝗲: 𝗧𝗵𝗲 𝗣𝗼𝘄𝗲𝗿 𝗼𝗳 𝗧𝗙-𝗜𝗗𝗙 𝗶𝗻 𝗔𝗜 In the journey to mastering Generative AI, it's essential to understand how we move from raw text to meaningful numerical data. We've explored foundational concepts like Bag-of-Words (BoW), which simply counts word frequency. But what if we could give more weight to words that truly matter? That's the purpose of TF-IDF (Term Frequency-Inverse Document Frequency), a cornerstone statistical method in NLP and Information Retrieval. My latest carousel post, "Key Topics to Learn in Gen AI: TF-IDF," explains this crucial technique in detail. TF-IDF is a numerical statistic used to evaluate how important a word is to a document within a larger collection of documents (a corpus). It’s a preprocessing and feature extraction technique that transforms raw text into numerical values, highlighting important terms while reducing the weight of common, less-meaningful words like "the" or "is." 𝗪𝗵𝘆 𝗧𝗙-𝗜𝗗𝗙 𝗥𝗲𝗺𝗮𝗶𝗻𝘀 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝘁 𝗧𝗼𝗱𝗮𝘆 Even though modern Generative AI models like GPT and BERT rely on sophisticated embeddings, TF-IDF is far from obsolete. It provides a valuable conceptual stepping stone and still has practical applications: Search Engines & Ranking: TF-IDF was, and in some variants still is, a central tool for search engines to rank documents by relevance. Text Summarization & Spam Filtering: It helps identify key sentences for summarization and highlights distinguishing terms for spam detection. Foundation for Modern AI: The core idea behind TF-IDF—giving attention to important words—inspired the attention mechanisms found in Transformer models. It is also used as a baseline for document retrieval in modern approaches like Retrieval-Augmented Generation (RAG). 𝗛𝗼𝘄 𝗜𝘁 𝗪𝗼𝗿𝗸𝘀: 𝗔 𝗦𝗶𝗺𝗽𝗹𝗲 𝗙𝗼𝗿𝗺𝘂𝗹𝗮 𝘄𝗶𝘁𝗵 𝗣𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 The TF-IDF score for a word in a document is the product of two parts: Term Frequency (TF): Measures how often a term appears in a document. Inverse Document Frequency (IDF): Measures how rare a term is across the entire corpus. Words that appear in many documents, like "the," receive a low IDF score, while rare and important words get a high score. By multiplying these two values, TF-IDF ensures that terms that are both frequent in a specific document and rare across the corpus receive a high score, effectively standing out as meaningful. By understanding TF-IDF, you gain a vital perspective on the evolution of NLP and the foundational principles that continue to influence the field of Generative AI. #GenerativeAI #NLP #TFIDF #MachineLearning #AI #DeepLearning #TextRepresentation #DataScience