• What is Data Mining (DM)? (30 minutes)
  • The data flood
  • Areas of applications
  • What can DM do? And how? And how well?
  • The Netflix Prize
• Two technical examples (45 minutes)
  • Wine classification
  • Direct mail response prediction
• DM and law enforcement (45 minutes)
  • Promises, including false ones
  • Total Information Awareness (2002-2003)
  • DM = Profiling?
• DM and Privacy (30 minutes)
  • Is DM a threat to privacy?
  • How can this threat be controlled?
  • Privacy preserving DM

Post your questions, observations, suggestions in the Comments area!

Definitions

• Wikipedia: Data Mining is the process of extracting hidden patterns from data.
• Federation of American Scientists: Data Mining involves the use of sophisticated data analysis tools to discover previously unknown, valid patterns and relationships in large data sets.
• From a blog I found with Google: Data Mining is a process of extraction of non-trivial patterns from massive datasets which either provides descriptive insights of the data (not perceived without this extraction) or provides actionable intelligence (in the form of reusable patterns which the process extracted).
• Greg Piatetsky-Shapiro: Data Mining is the process of finding new and potentially useful knowledge from data. Data Mining is the art and science of finding interesting and useful patterns in data.

Synonyms

• Knowledge discovery (KD)
• Machine learning
• Statistical learning

Websites

• KDNuggets
• ACM special interest group on KD and DM
• Netflix Prize

Intro Reading

• Herb Edelstein’s “Introduction to Data Mining and Knowledge Discovery”
  • Read pages 1-10.
  • Read anything that interests you from pages 11-21.

DM, Terrorism, Crime

• The Electronic Pricavy Information Center on TIA (Total Information Awareness). This program seems to be dead … but maybe not.
• Bruce Schneier on Mining for terrorists
• Herb Edelstein’s views on TIA
• Bruce Schneier on DM use by the police of Richmond, VA … with comments
• Fraud detection for credit cards, for electronic commerce

DM and Privacy

• Counterterrorism programs using DM should be evaluated for effectiveness and privacy impact – press release and summary of  NAS report (2008)
• A discussion of DM and privacy was initiated by DHS
• The view of the ACLU
• A letter from ACM on DM and privacy
• Privacy preserving data mining
• Privacy preserving publishing of social network data

Software and Datasets

• WEKA – free
• RapidMiner – free
• SAS Analytics
• Salford Systems
• UCI datasets
• KDD Cup annual competition