Lect 1 introduction

Dr. Hrudaya Kumar Tripathy
Introduction

 Definition, motivation & application
 Branches of data mining
 Classification, clustering,Association rule mining
 Some classification techniques

§ There has been enormous
d a t a g r o w t h i n b o t h
commercial and scientific
databases due to advances
in data generation and
collection technologies
§ New mantra
§ Gather whatever data
you can whenever and
wherever possible.
§ Expectations
§ Gathered data will have
value either for the
purpose collected or for
a purpose not envisioned. Computational Simulations
Social Networking:
Twitter
Sensor Networks
Traffic Patterns
Cyber Security E-Commerce

 Before 1600, empirical science
 1600-1950s, theoretical science
◦ Each discipline has grown a theoretical component.Theoretical models often
motivate experiments and generalize our understanding.
 1950s-1990s, computational science
◦ Over the last 50 years, most disciplines have grown a third, computational branch
(e.g. empirical, theoretical, and computational ecology, or physics, or linguistics.)
◦ Computational Science traditionally meant simulation. It grew out of our inability
to find closed-form solutions for complex mathematical models.
 1990-now, data science
◦ The flood of data from new scientific instruments and simulations
◦ The ability to economically store and manage petabytes of data online
◦ The Internet and computing Grid that makes all these archives universally
accessible
◦ Scientific info. management, acquisition, organization, query, and visualization tasks
scale almost linearly with data volumes. Data mining is a major new challenge!
 Jim Gray and Alex Szalay, TheWorldWideTelescope:An Archetype for Online Science,
Comm.ACM, 45(11): 50-54, Nov. 2002

• 1960s: Data collection, database creation, IMS and network DBMS
• 1970s: Relational data model, relational DBMS implementation
• 1980s: RDBMS, advanced data models (extended-relational, OO,
deductive, etc.), Application-oriented DBMS (spatial, scientific,
engineering, etc.)
• 1990s: Data mining, data warehousing, multimedia databases, and Web
databases
• 2000s
• Stream data management and mining
• Data mining and its applications
• Web technology (XML, data integration) and global information
systems

 Lots of data is being collected and
warehoused
◦ Web data
 Yahoo has Peta Bytes of web data
 Facebook has billions of active users
◦ e-commerce
• Amazon handles millions of visits/day
◦ purchases at department/
grocery stores
◦ Bank/Credit Card
transactions
 Computers have become cheaper and more powerful
 Competitive Pressure is Strong
◦ Provide better, customized services for an edge (e.g. in Customer
Relationship Management)

Sky Survey Data
 Data collected and stored at enormous
speeds (GB/hour)
◦ remote sensors on a satellite
 NASA EOSDIS archives over petabytes of earth
science data / year
◦ telescopes scanning the skies
 Sky survey data
◦ High-throughput biological data
◦ scientific simulations
 terabytes of data generated in a few hours
 Data mining helps scientists
◦ in automated analysis of massive datasets
◦ In hypothesis formation
SurfaceTemperature of Earth
Gene Expression Data

Improving health care and reducing costs
Finding alternative/ green energy sources
Predicting the impact of climate change
Reducing hunger and poverty by
increasing agriculture production

 Data explosion problem
◦ Automated data collection tools and mature database
technology lead to tremendous amounts of data stored
in databases, data warehouses and other information
repositories
 We are drowning in data, but starving for knowledge!
 Solution: Data warehousing and data mining
◦ Data warehousing and on-line analytical processing
◦ Extraction of interesting knowledge (rules, regularities,
patterns, constraints) from data in large databases

 Data mining (knowledge discovery in databases):
◦ Extraction of interesting (non-trivial, implicit,
previously unknown and potentially useful)
information or patterns from data in large databases.
 Alternative names and their “inside stories”:
◦ Data mining: a misnomer?
◦ Knowledge discovery(mining) in databases (KDD),
knowledge extraction, data/pattern analysis, data
archeology, business intelligence, etc.

 Many Definitions
◦ Exploration & analysis, by automatic or semi-automatic
means, of large quantities of data in order to discover
meaningful patterns

 Finding hidden information in a database
 Fit data to a model
 Similar terms
◦ Exploratory data analysis
◦ Data driven discovery
◦ Deductive learning

l What is not Data
Mining?
– Look up phone
number in phone
directory
– Q u e r y a W e b
search engine for
information about
“Amazon”
l What is Data Mining?
– Certain names are more
prevalent in c e r t a in U S
locations (O’Brien, O’Rurke,
O’Reilly… in Boston area)
– G ro u p t o g e t h e r s i m i l a r
documents returned by search
engine according to their
c o n t e x t ( e . g . A m a z o n
rainforest,Amazon.com,)

 Draws ideas from machine learning/AI, pattern recognition,
statistics, and database systems
 Traditional techniques may be unsuitable due to data that is
◦ Large-scale
◦ High dimensional
◦ Heterogeneous
◦ Complex
◦ Distributed
 A key component of the emerging field of data science and
data-driven discovery

Data
Mining
Statistics
OtherDiscipline
Visualization
Algorithm
Pattern
Recognition
Machine
Learning
Database
Technology

 Tremendous amount of data
◦ Algorithms must be highly scalable to handle such as tera-bytes of data
 High-dimensionality of data
◦ Micro-array may have tens of thousands of dimensions
 High complexity of data
◦ Data streams and sensor data
◦ Time-series data, temporal data, sequence data
◦ Structure data, graphs, social networks and multi-linked data
◦ Heterogeneous databases and legacy databases
◦ Spatial, spatiotemporal, multimedia, text and Web data
◦ Software programs, scientific simulations
 New and sophisticated applications

 This is a view from typical
database systems and data
warehousing communities
 Data mining plays an
e s s e n t i a l ro l e i n t h e
knowledge discover y
process
Data Cleaning
Data Integration
Databases
Data Warehouse
Task-relevant Data
Selection
Data Mining
Pattern Evaluation

The knowledge discovery process is depicted in Figure as
an iterative sequence of the following steps:
1. Data cleaning (to remove noise and inconsistent data)
2. Data integration (where multiple data sources may be combined)
3. Data selection (where data relevant to the analysis task are retrieved
from the database)
4. Data transformation (where data are transformed and consolidated
into forms appropriate for mining by performing summary or aggregation
operations)
5. Data mining (an essential process where intelligent methods are
applied in order to extract data patterns)
6. Pattern evaluation (to identify the truly interesting patterns
representing knowledge based on interestingness measures)
7. Knowledge presentation (where visualization and knowledge
representation techniques are used to present the mined knowledge to
the user)

Increasing potential
to support
business decisions End User
Business
Analyst
Data
Analyst
DBA
Decision
Making
Data Presentation
Visualization Techniques
Data Mining
Information Discovery
Data Exploration
Statistical Summary, Querying, and Reporting
Data Preprocessing/Integration, Data Warehouses
Data Sources
Paper, Files, Web documents, Scientific experiments, Database Systems

1. Identify the problem
2. Use data mining techniques to transform the
data into information
3.Act on the information
4. Measure the results

 Query
◦ Well defined
◦ SQL
 Query
◦ Poorly defined
◦ No precise query language
 Data
◦ Operational data
 Output
◦ Precise
◦ Subset of database
 Data
◦ Not operational data
 Output
◦ Fuzzy
◦ Not a subset of database

 Database
 Data Mining
– Find all customers who have purchased milk
– Find all items which are frequently purchased with milk. (Association rules)
– Find all credit applicants with last name of Smith.
– Identify customers who have purchased more than $10,000 in the last
month.
– Find all credit applicants who are poor credit risks. (Classification)
– Identify customers with similar buying habits. (Clustering)

 Data to be mined
◦ Database data (extended-relational, object-oriented, heterogeneous,
legacy), data warehouse, transactional data, stream, spatiotemporal,
time-series, sequence, text and web, multi-media, graphs & social and
information networks
 Knowledge to be mined (or: Data mining functions)
◦ Characterization, discrimination, association, classification, clustering,
trend/deviation, outlier analysis, etc.
◦ Descriptive vs. predictive data mining
◦ Multiple/integrated functions and mining at multiple levels
 Techniques utilized
◦ Data-intensive, data warehouse (OLAP), machine learning, statistics,
pattern recognition, visualization, high-performance, etc.
 Applications adapted
◦ Retail, telecommunication, banking, fraud analysis, bio-data mining, stock
market analysis, text mining,Web mining, etc.

 Database-oriented data sets and applications
◦ Relational database, data warehouse, transactional database
 Advanced data sets and advanced applications
◦ Data streams and sensor data
◦ Time-series data, temporal data, sequence data (incl. bio-sequences)
◦ Structure data, graphs, social networks and multi-linked data
◦ Object-relational databases
◦ Heterogeneous databases and legacy databases
◦ Spatial data and spatiotemporal data
◦ Multimedia database
◦ Text databases
◦ The World-Wide Web

 Decisions in data mining
◦ Kinds of databases to be mined
◦ Kinds of knowledge to be discovered
◦ Kinds of techniques utilized
◦ Kinds of applications adapted
 Data mining tasks
◦ Descriptive data mining
◦ Predictive data mining

 PredictionTasks (Predictive)
◦ The objective of these tasks is to predict the value of a particular
attribute based on the values of other attributes.
◦ The attribute to be predicted is commonly known as the target or
dependent variable, while the attributes used for making the
prediction are known as the explanatory or independent
variables.
 DescriptionTasks (Descriptive)
◦ Find human-interpretable patterns that describe the data.
• Common data mining tasks
 Classification [Predictive]
 Clustering [Descriptive]
 Association Rule Discovery [Descriptive]
 Sequential Pattern Discovery [Descriptive]
 Regression [Predictive]
 Deviation Detection [Predictive]

Tid Refund Marital
Status
Taxable
Income Cheat
1 Yes Single 125K No
2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
12 Yes Divorced 220K No
10
Predictive M
odeling
Clustering
Association
Rules
Anom
aly
Detection
Milk
Data
Data Mining Tasks …

29
 Information integration and data warehouse
construction
◦ Data cleaning, transformation, integration, and
multidimensional data model
 Data cube technology
◦ Scalable methods for computing (i.e., materializing)
multidimensional aggregates
◦ OLAP (online analytical processing)
 Multidimensional concept description:
Characterization and discrimination
◦ Generalize, summarize, and contrast data characteristics, e.g.,
dry vs. wet region

30
 Frequent patterns (or frequent itemsets)
◦ What items are frequently purchased together in your
Walmart?
 Association, correlation vs. causality
◦ A typical association rule
 Diaper  Beer [0.5%, 75%] (support, confidence)
 Tea  Sugar [0.5%, 75%] (support, confidence)
◦ Are strongly associated items also strongly correlated?
◦ How to mine such patterns and rules efficiently in large
datasets?
◦ How to use such patterns for classification, clustering, and
other applications?

31
 Classification and label prediction
◦ Construct models (functions) based on some training examples
◦ Describe and distinguish classes or concepts for future
prediction
 E.g., classify countries based on (climate), or classify cars based
on (gas mileage)
◦ Predict some unknown class labels
 Typical methods
◦ Decision trees, naïve Bayesian classification, support vector
machines, neural networks, rule-based classification, pattern-
based classification, logistic regression, …
 Typical applications:
◦ Credit card fraud detection, direct marketing, classifying stars,
diseases, web-pages, …

32
 Unsupervised learning (i.e., Class label is unknown)
 Group data to form new categories (i.e., clusters), e.g.,
cluster houses to find distribution patterns
 Principle: Maximizing intra-class similarity & minimizing
interclass similarity
 Many methods and applications

33
 Outlier analysis
◦ Outlier:A data object that does not comply with the
general behavior of the data
◦ Noise or exception? ― One person’s garbage could be
another person’s treasure
◦ Methods: by product of clustering or regression
analysis, …
◦ Useful in fraud detection, rare events analysis

34
 Sequence, trend and evolution analysis
◦ Trend, time-series, and deviation analysis: e.g., regression
and value prediction
◦ Sequential pattern mining
 e.g., first buy digital camera, then buy large SD memory
cards
◦ Periodicity analysis
◦ Motifs and biological sequence analysis
 Approximate and consecutive motifs
◦ Similarity-based analysis
 Mining data streams
◦ Ordered, time-varying, potentially infinite, data streams

35
 Graph mining
◦ Finding frequent subgraphs (e.g., chemical compounds), trees (XML),
substructures (web fragments)
 Information network analysis
◦ Social networks: actors (objects, nodes) and relationships (edges)
 e.g., author networks in CS, terrorist networks
◦ Multiple heterogeneous networks
 A person could be multiple information networks: friends, family,
classmates, …
◦ Links carry a lot of semantic information: Link mining
 Web mining
◦ Web is a big information network: from PageRank to Google
◦ Analysis of Web information networks
 Web community discovery, opinion mining, usage mining, …

36
 Are all mined knowledge interesting?
◦ One can mine tremendous amount of “patterns” and
knowledge
◦ Some may fit only certain dimension space (time,
location, …)
◦ Some may not be representative, may be transient, …
 Evaluation of mined knowledge → directly mine only
interesting knowledge?
◦ Descriptive vs. predictive
◦ Coverage
◦ Typicality vs. novelty
◦ Accuracy
◦ Timeliness
◦ …

 Mining Methodology
◦ Mining various and new kinds of knowledge
◦ Mining knowledge in multi-dimensional space
◦ Data mining:An interdisciplinary effort
◦ Boosting the power of discovery in a networked environment
◦ Handling noise, uncertainty, and incompleteness of data
◦ Pattern evaluation and pattern- or constraint-guided mining
 User Interaction
◦ Interactive mining
◦ Incorporation of background knowledge
◦ Presentation and visualization of data mining results

 Efficiency and Scalability
‒ Efficiency and scalability of data mining algorithms
‒ Parallel, distributed, stream, and incremental mining methods
 Diversity of data types
‒ Handling complex types of data
‒ Mining dynamic, networked, and global data repositories
 Data mining and society
‒ Social impacts of data mining
‒ Privacy-preserving data mining
‒ Invisible data mining

 Web page analysis: from web page classification, clustering to
PageRank & HITS algorithms
 Collaborative analysis & recommender systems
 Basket data analysis to targeted marketing
 Biological and medical data analysis: classification, cluster analysis
(microarray data analysis), biological sequence analysis, biological
network analysis
 Data mining and software engineering (e.g., IEEE Computer,Aug. 2009
issue)
 From major dedicated data mining systems/tools (e.g., SAS, MS SQL-
Server Analysis Manager, Oracle Data Mining Tools) to invisible data
mining

Lect 1 introduction

More Related Content

What's hot

Similar to Lect 1 introduction

More from hktripathy

Recently uploaded

Lect 1 introduction