Downloaded 735 times
More Related Content
Similar to Big Data Analysis Patterns with Hadoop, Mahout and Solr
![[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-buildingaisystemsthatusersandcompanieslove-251124030845-038f7732-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Mobile Development] Crafting Immersive UI with E2E and AGSL Shade...](https://cdn.slidesharecdn.com/ss_thumbnails/md-craftingimmersiveuiwithe2eandagslshaderveronicaputrianggraini-251124030840-0c677f44-thumbnail.jpg?width=640&height=640&fit=bounds)
Big Data Analysis Patterns with Hadoop, Mahout and Solr
- 1. Big Data Analysis Patterns AtlantaBig Data User Group 8/15/2013 1
- 2. whoami • Brad Anderson • Solutions Architectat MapR (Atlanta) • ATLHUG co-chair • NoSQL East Conference 2009 • “boorad” most places (twitter, github) • banderson@maprtech.com 2
- 3. Announcements Next ATLHUG Meeting- Sept. 26 – How Google Does Big Data Wednesday – MapR Data Warehouse Offload Roadshow MapR Upcoming Training • • • 3 MapR M7 & HBase for Developers on August 27 in Campbell, CA MapR M7 & HBase for Developers on Sept 17 in Reston, VA MapR M5 for Administrators on Oct 3 in Campbell, CA 3
- 4.
- 5.
- 6. Big Data isnot new! but the tools are. 6
- 7. The Good Newsin Big Data: “Simple algorithms and lots of data trump complex models” Halevy, Norvig, and Pereira, Google IEEE Intelligent Systems 7
- 8. The Challenge: SoMany Solutions! What solutions fit your business problem? For example, do you need… Apache Mahout? Storm? Apache Solr/Lucene? Apache HBase (or MapR M7)? Apache Drill (or Impala?) d3.js or Tableau? Node.js 8 Apache Hadoop? Titan? 8
- 9. Ask a DifferentQuestion It may be more useful to better define the problem by asking some of these questions: How large is the data to be queried? (the analysis volume) What time frame is appropriate for your query response? How fast is data arriving? (bursts or continuously?) Are queries by sophisticated users? Are you looking for common patterns or outliers? 9 How large is the data to be stored? How are your data sources structures? 9
- 10. Picking the BestSolution Your responses to these questions can help you better: define the problem recognize the analysis pattern to which it belongs guide the choice of solutions to try But first, here’s a quick review of a few of the technologies you might choose, and then we will focus on three of the questions as a part of the landscape. 10 10
- 11. Apache Solr/Lucene Solr/Lucene isa powerful search engine used for flexible, heavily indexed queries including data such as Full text Geographical data Statistically weighted data Solr is a small data tool that has flourished in a big data world 11
- 12. Apache Mahout Mahout providesa library of scalable machine learning algorithms useful for big data analysis based on Hadoop or other storage systems. Mahout algorithms mainly are used for Recommendation (collaborative filtering) Clustering Classification Mahout can be used in conjunction with solutions such as Solr: You might use Mahout to create a co-occurrence data base that could then be queried using a search tool such as Solr 12
- 13. Apache Drill Google Dremelclone Pluggable Query Languages – – Pluggable Storage Backends – – – Starts with ANSI SQL 2003 Hive, Pig, Cascading, MongoQL, … Hadoop, Hbase MongoDB (BSON) RDBMS? Bypasses MapReduce 13
- 14. Storm Realtime Stream ComputationEngine Horizontal Scalability Guaranteed Data Processing Fault Tolerance Higher level abstraction over: – – Message Queues Worker Logic “The Hadoop of Realtime” 14
- 15. Titan Distributed Graph Database PropertyGraph Pluggable Backend Storage – – – Search Integrated – – Solr/Lucene Elastic Search Faunus – HBase or M7 Cassandra Berkeley DB Batch processing of large graphs Fulgora – – Graph traversals on subset In-memory 15
- 16. Using the Answersto Guide Your Choices For simplicity, let’s focus in on the first three questions: How large is the data to be stored? How large is the data to be queried? (the analysis volume) What time frame is appropriate for your query response? 16
- 17. Big Data DecisionTree How big is your data? <10 GB mid ? ? A Single element at a time >200 GB What size queries? One pass over 100% B Response time? C Big storage Multiple passes over big chunks Streaming < 100s (human scale) D 17 throughput not response E
- 18. Use Cases Company DataShape Technique(s) Business Value 18
- 19.
- 20.
- 21.
- 22. Telecommunications Data Shape Lots ofData Lots of Queries across Large Sets Throughput important 22
- 23.
- 24.
- 25.
- 26.
- 27. Credit Card Issuer Data Shape CustomerPurchase History (big) Merchant Designations Merchant Special Offers Throughput important Recommendations 27
- 28. Credit Card Issuer Techniques A RecommendationEngine with Mahout and Solr/Lucene History matrix One row per user One column per thing 28
- 29. Credit Card Issuer Techniques Recommendation basedon cooccurrence Cooccurrence gives item-item mapping One row and column per thing 29
- 30. Credit Card Issuer Techniques Cooccurrence matrixcan also be implemented as a search index 30
- 31.
- 32. Credit Card Issuer Techniques User history SolR SolR Indexer Solr Indexer search Web tier 8Hrs 3 Min Item metadata Index shards 32
- 33. Credit Card Issuer Techniques Hadoop Purchase History Export (4 hrs) App App Merchant Information Recommendation EngineResults (Mahout) Presentation Data Store (DB2) App App Merchant Offers App Import (4 hrs) 33
- 34. Credit Card Issuer Techniques Hadoop Purchase History Merchant Information Recommendation Engine Results (Mahout) Index Update (3min) App App Recommendation Search Index (Solr) App App Merchant Offers App 34
- 35.
- 36. Waste & RecyclingLeader Idle Alerts 36
- 37. Data Shape Truck GeolocationData – 20,000 trucks – 5 sec interval (arriving quickly) Landfill Geographic Boundaries 37
- 38. Techniques Realtime Stream Computation (Storm) Truck Geolocation Data Hadoop Storage Immediate Alerts BatchComputation (MapReduce) Tax Reduction Reporting Shortest Path Graph Algorithm (Titan) Route Optimization 38
- 39.
- 40. Beverage Company Social EngagementApplication 40
- 41. Data Shape Tweets, FBMessages Person, Activity links Graph Traversal 41
- 42. Consumer Activity Graph Wal*Mart.com Ebay Shopping.com Sam’s EbayMotors Dollar General StubHub CVS 42 Toys R Us
- 43. Techniques Property Graph (Titan) Social Activity Stream Key/Value Store (MapRM7) 43 Graph Traversal (Faunus/Fulgora)
- 44.
- 45.
- 46. Data Sources Anti-Money Laundering ConsumerTransactions 46
- 47.
- 48. Techniques AML Data Lake (Hadoop) Suspicious Events Consumer Transactions Analyst Latent DirichletAllocation, Bayesian Learning Neural Network, Peer Group Analysis 48
- 49.
- 50. Machine Learning Search Relevance DNAMatching 50
- 51. Data Sources Birth, Death,Census, Military, I mmigration records Search Behavior Activity DNA SNP (snips) 51
- 52. Techniques Record Linking SearchRelevance Clickstream Behavior Security Forensics DNA Matching 52
- 53.
- 54.
- 55. Data Sources Inrix RoadSegment Data Avg Speed / minute / segment – Reference Speeds – Road Segment Geolocation Data 55
- 56. Techniques Bottleneck DetectionAlgorithm Time Offset Correlations – Alternate Routes Predictive Congestion Analysis – Growth & Term Assumptions 56
- 57.
- 58.
- 59.
- 60. Similar Characteristics Lots ofData Structured, Semi-Structured, Unstructured Varied Systems Interoperating – Hadoop, Storm, Solr, MPP, Visualizations Increase Revenue Decrease Costs 60
- 61.