Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 1: Introduction and Resources
Introduction and Resources
Welcome to my 7-post series on Complex Event Processing with Mircosoft StreamInsight. Enjoy!
Microsoft StreamInsight is a very lightweight, powerful and for .NET developers easy to use Complex Event Processing (CEP) engine. It uses .NET LINQ queries to write queries on data streams.
With minimal effort, StreamInsight allows you to answer questions such as “what is the one minute aggregate of my value X grouped by dimension D”. This series of blog posts explains the concepts behind CEP and StreamInsight and implements a sample application that can be used to analyze live data from Twitter.
Index of this blog series:
Part 1 (03.10.13): CEP with StreamInsight and Twitter 1:
Introduction and Resources
Part 2 (06.10.13): CEP with StreamInsight and Twitter 2:
What are CEP and StreamInsight
Part 3 (13.10.13): CEP with StreamInsight and Twitter 3:
Part 4 (20.10.13): CEP with StreamInsight and Twitter 4:
Part 5 (27.10.13): CEP with StreamInsight and Twitter 5:
The Sample Application 1
Part 6 (03.11.13): CEP with StreamInsight and Twitter 6:
The Sample Application 2
Part 7 (10.11.13): CEP with StreamInsight and Twitter 7:
The Sample Application 3
The best resources that I found:
Video: “What’s new in Complex Event Processing using StreamInsight for SqlServer 2.1” by Torsten Grabs, Principal Program Manager Lead, Microsoft
Video: “My Campus Walk: I walk, StreamInsight analyzes, Worldwide Telescope visualizes.” Interesting project about visualization of a walk path from geo-spatial data processed by StreamInsight.
Video: “Heat mapping H5N1 Outbreaks with StreamInsight and Binq Maps”. Really awesome demo application that feeds processed stream data to a Bing Maps application.
A tool I absolutely recommend, especially for StreamInsight Development, is LINQPad (http://www.linqpad.net/). LINQPad includes the following StreamInsight sample packages that can be executed and played with:
- 101 StreamInsight Samples
- Hitchhikers Guide to StreamInsight
- Performance Counter Samples
- Reactive Extensions
- LINQ in Action
If you use the free version of LINQPad you will not get any intellisense or auto-complete functionality. In my opinion this makes it useless. As soon as I bought a personal license (about USD30), LINQPad turned into one of my favorite dev tools of all times. It is extremely lightweight and understands C#, VB, F#, SQL, LINQ, ADO.NET, StreamInsight, etc.
TwitterStream Sample: The application that we are going to develop in the following blog posts:
Hitchhikers Guide to StreamInsight Queries: Very comprehensive guide to writing queries in StreamInsight. Code is included in the codeplex download below. Samples are included in LINQPad.
StreamInsight Product Team Samples 2.1
StreamInsight Product Team Samples 2.0
Very important collection of samples such as: TextFileReaderAdapter, TextFileWriterAdapter, SqlAdapter, WcfAdapter and lots of applications.
JAhlen StreamInsight Example Adapters v2.1
Stock trading sample application.
Blog: Official Blog of the Microsoft StreamInsight Dev Team
Blog: StreamInsight explained to non-programmers
Blog: Getting Started with StreamInsight 2.1
Technet: Microsoft StreamInsight
MSDN: Microsoft StreamInsight
MSDN Forum: SQL Server –> Microsoft StreamInsight