ToolTip: Decoding Base64 Images with Chrome Data URL

Reading the great blog post about Base64 encoding and decoding in .NET/C# written by Jerry Nixon (http://blog.jerrynixon.com/2014/11/reading-and-writing-base64-in-windows.html), I immediately remembered a great trick that comes in very handy when using Base64 encoding.

Suppose you have an image that you would like to encode into base64:

sn

You will end up with a very long string, such as this one:

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBKRXhpZgAATU0AKgAAAAgAAwEaAAUAAAABAAAA MgEbAAUAAAABAAAAOgEoAAMAAAABAAIAAAAAAAA5OHAAAJiWgDk4cAAAmJaA/+EF7Gh 0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8APD94cGFja2V0IGJlZ2luPSfvu78nIGlkPSd XNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQnPz4NCjx4OnhtcG1ldGEgeG1sbnM6eD0iYW RvYmU6bnM6bWV0YS8iPjxyZGY6UkRGIHhtbG…

During testing of your application, it can be really painful to convert these strings back into jpg files just to see if nothing got missing or corrupted. You could either write a small program, that decodes the base64 string, use a tool or you could use an online base64 decoder/encoder such as this one:

http://www.freeformatter.com/base64-encoder.html

Aaaand, here comes the trick:

If you have Google Chrome installed, you can use a feature called Data Uri to decode your Base64 string. Just type into the chrome address bar:

data:image/jpeg;base64,<base64>

where <base64> is the base64 encoded data. In our example:

data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4QBKRXhpZU0…

And BOOM, chrome decodes the Base64 data and displays our image!

image

You can even specify a different target type, such as:

data:image/png;base64,<base64>

or

data:image/webp;base64,<base64>

I like this trick very much. It is a huge timesaver! I hope you like it too!

Category: Business  Tags: , , ,  Leave a Comment

Microsoft TechTalk: S.O.L.I.D. Principles and DDD

image

On September 18, 2014, I held a Tech Talk at the Microsoft Switzerland Offices in Walisellen. The title was “Enterprise Software Architecture and Domain-Driven Design”. The main goal of the talk was to explain that with the .NET Framework getting more and more mature, it becomes ever more important to think about how we deal with legacy .NET code, that is existing .NET code.

Everyday, more and more .NET projects will not start from scratch but start on an already existing code base. Therefore, we as developers need to understand how to reduce our legacy by writing more maintainable code.

The SOLID Principles are an approach on how to write modular code that enables reuse scenarios. Robert C. Martin, aka. Uncle Bob first came up with the principles in the early 2000s.

image

The SOLID principles are:

  • SRP: Single Responsibility Principle
  • OCP: Open/Closed Principle
  • LSP: Liskov Substitution Principle
  • ISP: Interface Segregation Principle
  • DIP: Dependency Inversion Principle

Each of the principles describes an approach to “correctly do object oriented programming (OOP)”.

After I explained the SOLID Principles and showed a couple of demos, we took a look at Domain Driven Design (DDD). DDD is an approach to designing a software solution by following a couple of recommendations.

image

It concentrates on creating a common understanding between the business experts and the developers by creating artifacts such as a common language (the ubiquitous language) and establishing a common model (the domain model), which is a graphical representation of the business domain. Using several DDD techniques, the model is iteratively refined by making it more correct and reducing complexity. A very important concept is to make implicit stuff explicit. Often things that seem simple or even trivial to the developer are absolutely crucial to the business. Therefore, it is vital to give these concepts a name and add them to the model as a visible artifact.

image

Using further techniques such as entities, aggregates, value objects, factories or repositories, the model is then translated into code.

Attached is the slide deck that I used during the presentation.

A big thank you goes out to Microsoft Switzerland for giving me the opportunity and to the engaged crowd for listening and thinking along!

Category: Business  Tags: , , ,  Leave a Comment

WPF: The Simplest Way to Get the Default Template of a Control as XAML

Every WPF Developer has probably faced this issue before. You want to change the default template of a control and need it in XAML. Where can you get the default template? There are multiple ways:

  1. Find the correct MSDN page for your framework version
  2. Use a tool
  3. Use Expression Blend

In my opinion, none of these solutions is very convenient. I always end up either not finding the correct version, not having blend installed, etc…

Alas, there is a much easier way. You can use the XamlWriter class to serialize the default template of your control:

1 public static void SaveDefaultTemplate() 2 { 3 var control = Application.Current.FindResource(typeof(ProgressBar)); 4 using (XmlTextWriter writer = new XmlTextWriter(@"defaultTemplate.xml", System.Text.Encoding.UTF8)) 5 { 6 writer.Formatting = Formatting.Indented; 7 XamlWriter.Save(control, writer); 8 } 9 }

This will produce the following file for the ProgressBar in the sample code:

1 <Style TargetType="Button" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib"> 2 <Style.BasedOn> 3 <Style TargetType="ButtonBase"> 4 <Style.Resources> 5 <ResourceDictionary /> 6 </Style.Resources> 7 <Setter Property="FrameworkElement.FocusVisualStyle"> 8 <Setter.Value> 9 <Style TargetType="IFrameworkInputElement"> 10 <Style.Resources> 11 <ResourceDictionary /> 12 </Style.Resources> 13 <Setter Property="Control.Template"> 14 <Setter.Value> 15 <ControlTemplate> 16 <Rectangle Stroke="{DynamicResource {x:Static SystemColors.ControlTextBrushKey}}" StrokeThickness="1" StrokeDashArray="1 2" Margin="2,2,2,2" SnapsToDevicePixels="True" /> 17 </ControlTemplate> 18 </Setter.Value> 19 </Setter> 20 </Style> 21 </Setter.Value> 22 </Setter> 23 <Setter Property="Panel.Background"> 24 <Setter.Value> 25 <SolidColorBrush>#FFDDDDDD</SolidColorBrush> 26 </Setter.Value> 27 </Setter> 28 <Setter Property="Border.BorderBrush"> 29 <Setter.Value> 30 <SolidColorBrush>#FF707070</SolidColorBrush> 31 </Setter.Value> 32 </Setter> 33 <Setter Property="TextElement.Foreground"> 34 <Setter.Value> 35 <DynamicResource ResourceKey="{x:Static SystemColors.ControlTextBrushKey}" /> 36 </Setter.Value> 37 </Setter> 38 <Setter Property="Border.BorderThickness"> 39 <Setter.Value> 40 <Thickness>1,1,1,1</Thickness> 41 </Setter.Value> 42 </Setter> 43 <Setter Property="Control.HorizontalContentAlignment"> 44 <Setter.Value> 45 <x:Static Member="HorizontalAlignment.Center" /> 46 </Setter.Value> 47 </Setter> 48 <Setter Property="Control.VerticalContentAlignment"> 49 <Setter.Value> 50 <x:Static Member="VerticalAlignment.Center" /> 51 </Setter.Value> 52 </Setter> 53 <Setter Property="Control.Padding"> 54 <Setter.Value> 55 <Thickness>1,1,1,1</Thickness> 56 </Setter.Value> 57 </Setter> 58 <Setter Property="Control.Template"> 59 <Setter.Value> 60 <ControlTemplate TargetType="ButtonBase"> 61 <Border BorderThickness="{TemplateBinding Border.BorderThickness}" BorderBrush="{TemplateBinding Border.BorderBrush}" Background="{TemplateBinding Panel.Background}" Name="border" SnapsToDevicePixels="True"> 62 <ContentPresenter RecognizesAccessKey="True" Content="{TemplateBinding ContentControl.Content}" ContentTemplate="{TemplateBinding ContentControl.ContentTemplate}" ContentStringFormat="{TemplateBinding ContentControl.ContentStringFormat}" Name="contentPresenter" Margin="{TemplateBinding Control.Padding}" HorizontalAlignment="{TemplateBinding Control.HorizontalContentAlignment}" VerticalAlignment="{TemplateBinding Control.VerticalContentAlignment}" SnapsToDevicePixels="{TemplateBinding UIElement.SnapsToDevicePixels}" Focusable="False" /> 63 </Border> 64 <ControlTemplate.Triggers> 65 <Trigger Property="Button.IsDefaulted"> 66 <Setter Property="Border.BorderBrush" TargetName="border"> 67 <Setter.Value> 68 <DynamicResource ResourceKey="{x:Static SystemColors.HighlightBrushKey}" /> 69 </Setter.Value> 70 </Setter> 71 <Trigger.Value> 72 <s:Boolean>True</s:Boolean> 73 </Trigger.Value> 74 </Trigger> 75 <Trigger Property="UIElement.IsMouseOver"> 76 <Setter Property="Panel.Background" TargetName="border"> 77 <Setter.Value> 78 <SolidColorBrush>#FFBEE6FD</SolidColorBrush> 79 </Setter.Value> 80 </Setter> 81 <Setter Property="Border.BorderBrush" TargetName="border"> 82 <Setter.Value> 83 <SolidColorBrush>#FF3C7FB1</SolidColorBrush> 84 </Setter.Value> 85 </Setter> 86 <Trigger.Value> 87 <s:Boolean>True</s:Boolean> 88 </Trigger.Value> 89 </Trigger> 90 <Trigger Property="ButtonBase.IsPressed"> 91 <Setter Property="Panel.Background" TargetName="border"> 92 <Setter.Value> 93 <SolidColorBrush>#FFC4E5F6</SolidColorBrush> 94 </Setter.Value> 95 </Setter> 96 <Setter Property="Border.BorderBrush" TargetName="border"> 97 <Setter.Value> 98 <SolidColorBrush>#FF2C628B</SolidColorBrush> 99 </Setter.Value> 100 </Setter> 101 <Trigger.Value> 102 <s:Boolean>True</s:Boolean> 103 </Trigger.Value> 104 </Trigger> 105 <Trigger Property="ToggleButton.IsChecked"> 106 <Setter Property="Panel.Background" TargetName="border"> 107 <Setter.Value> 108 <SolidColorBrush>#FFBCDDEE</SolidColorBrush> 109 </Setter.Value> 110 </Setter> 111 <Setter Property="Border.BorderBrush" TargetName="border"> 112 <Setter.Value> 113 <SolidColorBrush>#FF245A83</SolidColorBrush> 114 </Setter.Value> 115 </Setter> 116 <Trigger.Value> 117 <s:Boolean>True</s:Boolean> 118 </Trigger.Value> 119 </Trigger> 120 <Trigger Property="UIElement.IsEnabled"> 121 <Setter Property="Panel.Background" TargetName="border"> 122 <Setter.Value> 123 <SolidColorBrush>#FFF4F4F4</SolidColorBrush> 124 </Setter.Value> 125 </Setter> 126 <Setter Property="Border.BorderBrush" TargetName="border"> 127 <Setter.Value> 128 <SolidColorBrush>#FFADB2B5</SolidColorBrush> 129 </Setter.Value> 130 </Setter> 131 <Setter Property="TextElement.Foreground" TargetName="contentPresenter"> 132 <Setter.Value> 133 <SolidColorBrush>#FF838383</SolidColorBrush> 134 </Setter.Value> 135 </Setter> 136 <Trigger.Value> 137 <s:Boolean>False</s:Boolean> 138 </Trigger.Value> 139 </Trigger> 140 </ControlTemplate.Triggers> 141 </ControlTemplate> 142 </Setter.Value> 143 </Setter> 144 </Style> 145 </Style.BasedOn> 146 <Style.Resources> 147 <ResourceDictionary /> 148 </Style.Resources> 149 </Style>

Category: Business  Tags: , ,  2 Comments

AzureAdventures: Setting up the Adventureworks Database in under 10 Minutes

This blog post will show you how easy it is to set up the famous Adventureworks database using the Microsoft Azure Cloud.

Before we start, a little background:

A couple of days ago, I was at a customer site and needed a simple database in order to show a .NET framework demo. I wanted to demonstrate an end-to-end sample that consisted of loading data from an SqlServer database by means of the Entity Framework (EF) and hooking the loaded data on a Windows Presentation Foundation (WPF) front end. The requirement for the database were really simple. I just needed a DB that would run on Microsoft SqlServer, contain a couple of tables with relations and some data in them.

I remembered the good old Adventureworks database that is a perfect fit for any database related product demo. There are various possibilities on how to get the Database on http://www.codeplex.com.

I decided to go for the out-of-the-box-ready solution that provides an *.mdf and an *.ldf file that can be attached to an Sql Server, using the following link: http://adventureworksdw2008.codeplex.com/

However, as usual, it was not as easy as I hoped it would be. :-(

When I tried to start the SqlServer on my box, using the SqlServer Configuration Manager, I got an error message stating that “The remote procedure call failed. [0x800706be].

image

It looked like my Sql Server installation was somehow broken. I fiddled around for about 20 minutes and decided that I would just reinstall the SqlServer and not waist more time. Unfortunately, the SqlServer is a piece of software that, in my humble opinion, is not installed in 5 minutes time. As soon as the installer completed, it triumphantly told me that everything installed perfectly EXCEPT FOR THE DATABASE ENGINE!!! *grrrrrr*. That surely pissed me off. Annoyed and disappointed I went to sleep. (Needless to say, it was already very late at night.)

That left me in the awkward situation that I would have to show up at the customer site in the morning and fix my database problem sometime between the getting out of bed and the start of the course that I was about to teach. *sigh*

But then, IT HIT ME!

Since I am a huge fan of Microsoft Azure and deeply appreciate the technology behind it and how it facilitates a developers life, I turned my back on the local SqlServer installation on my notebook and looked up into the skies or, I must say, the cloud. And, alas, the Azure cloud really saved my day!!! Literally 5 minutes later, I had my db running and could even take a coffee break before the course started. *Tataaa*.

In the next section, I will demonstrate how amazingly easy it is to get the Adventureworks db running on Microsoft Azure. Keep in mind that the process that I am going to describe replaces all of these tasks:

  • Providing a virtual or physical machine
  • Downloading an SqlServer Installer
  • Organizing a License
  • Installing the SqlServer
  • Configuring the SqlServer
  • Downloading and Installing Adventureworks

 

Running Adventureworks on Microsoft Azure SQL

The following tutorial is divided into four parts:

1. Prerequisites
2. Provisioning an Azure Sql Database Server
3. Getting and Deploying the Scripts
4. Verifying the Installation

Let’s get started:

Step 1: The Prerequisites

You need 3 things in order to reproduce this example:

1. A Microsoft Azure Subscription
If you do not have one, you can get a free trial subscription for one month and 200 CHF at:
http://azure.microsoft.com/en-us/pricing/free-trial/
2. A computer with an installation of the .NET framework in version 4.
If you run Windows 8, you are set. If you run an earlier version and .NET 4.0 is not installed, you can get it at the Microsoft Download Center: http://msdn.microsoft.com/en-us/vstudio/aa496123
3. The Azure db must be accessible from your client computer. This is a configuration setting in the Azure Management Dashboard. Details can be found below.

Note that currently there are two Azure Portals. The old one under https://manage.windowsazure.com and the new one under https://portal.azure.com. For simplicity, I am going to use the old one.

Step 2: Provisioning an Azure Sql Database Server (Old Portal/manage.windowsazure.com)

After you have created your Microsoft Azure account, go to the management portal: http://manage.windowsazure.com and log in.

On the portal, select SQL DATABASES from the navigation on the left hand side:

image

Once you are on the SQL DATABASES page, click SERVERS on the top navigation:

image

Finally, click the ADD button at the bottom:

image

The CREATE SERVER wizard will pop up, fill in the fields and confirm. Select a region that is close to your location.

image

After some time, it took about 30 seconds for me, your database server is ready to go! Click the little arrow on the right of the server name to proceed to the server settings:

image

On the following page, click CONFIGURE and then click the arrow on the right, where it says “ADD TO THE ALLOWED IP ADDRESSES”. This configuration is vital since it allows your local machine, where the browser is running, to access the database server. IMPORTANT: Do not forget to hit the save button, once completed!

image

Note the server name and the credentials that you specified and we can go on with the next step. Alternatively, you can navigate to the DASHBOARD page and copy the MANAGE URL from there.

Step 3: Getting and deploying the scripts

Navigate to http://msftdbprodsamples.codeplex.com/releases/view/37304 and download the “AdventureWorks2012ForWindowsAzureSqlDatabase” file. Save and extract the zip file.

Open a CMD prompt with administrative privileges and navigate to the directory where you unzipped the package and there into the \AdventureWorks directory.

Enter the following command:

CreateAdventureWorksForSqlAzure.cmd <servername> <username> <password>

Important Hints!
Make sure you enter the name as “username@servername”. Providing the username only can lead to problems during the installation. Additionally, you have to use the full server name, as in: zvnhp88skk.database.windows.net.
Example:

CreateAdventureWorksForSqlAzure.cmd zvnhp88skk.database.windows.net mme@zvnhp88skk.database.windows.net mypassword

image 
After roughly 5 minutes, the script should show “Installation Completed”. That’s it, your database is up and running.

Step 4: Verifying the installation

You can use any SqlServer tool, such as Visual Studio or the SqlServer Management Studio to connect to your DB and verify that the data is there. I would like to show a very simple approach using SqlCmd.exe. SqlCmd is shipped with Visual Studio and the easiest way to use sqlcmd.exe is by using the Visual Studio Command Prompt. Run the following command inside the VS command prompt:

sqlcmd.exe –S <servername> –d <db name> –U <username> –q <query>

or, in our case:

sqlcmd.exe –S zvnhp88skk.database.windows.net –d AdventureWorks2012 –U mme@zvnhp88skk –q “select top 10 firstname, lastname from person.person”

If you see 10 records that contain names, the data is there and we are done!

Important hint: Always keep in mind that Microsoft Azure is based on a pay-as-you go policy and you will be charged on a per-minute base for your database instance! If you don’t need it, shut it down!

Getting started with Orchard CMS – The Modules and Features you need

After having installed Orchard as shown in http://www.12qw.ch/2014/03/getting-started-with-orchard-cms-installing-orchard it is very important to know what the first modules are that need to be installed. In this post I will give you an overview about the most important modules/features and why you need them.

The following features help you with questions such as:

  • How can I get Edit buttons on my Orchard pages?
  • How can I influence my Orchard Layout?
  • How can I get table editing functionality in my Orchard HTML Editor.

Widget Control Wrapper

The widget control wrapper is installed by default. Once activated, it will render “Edit” Buttons for all the content sections on the front page. Therefore this:

image

becomes this:

image

Follow these steps to activate the Widget Control Wrapper:

  • Log on to your web site
  • Click the dashboard link at the bottom
  • Click Modules in the navigation
  • Under Features search for Widget
  • Click enable on the Widget Control Wrapper Box

Shape Tracing

The shape tracing feature creates a menu on the web site that allows you to look at the layout structure of the page. Once activated, it displays a small icon at the bottom of the page:

image

The icon opens a menu that lets you analyze all the content element and look at their configuration in terms of Shape, Model, Placement, Template and HTML.

image

TinyMCEDeluxe

TinyMCEDeluxe extends the built-in WYSIWYG HTML Editor with more functionality, such as table editing features. So this:

image

Becomes this:

image 

TinyMCEDeuluxe is a module that needs to be installed from the gallery. In order to do this, navigate to Modules –> Gallery. Search for TinyMCEDeluxe and install the module.

IMPORTANT: Before you activate TinyMCEDeluxe, you MUST deactivate the built-in editor “TinyMCE”.

Getting started with Orchard CMS – Installing Orchard

Orchard CMS is a free Content Management System (CMS) based on Microsoft ASP.NET MVC. http://www.orchardproject.net/

Orchard is very powerful due to its rich extensions in the form of Modules (= Feature Packages) and Themes. It is highly customizable due to its full .NET integration and extensibility.

Orchard can be installed either by downloading with the Web Platform installer or by using WebMatrix. I am going to show you a brief description on how to create an orchard website using WebMatrix.

1. Installing WebMatrix

Microsoft WebMatrix is a free Microsoft web development tool. It focuses on easy creation, configuration, publication and maintenance of websites. The power of WebMatrix lies in the simplicity of its usage and, to my opinion, in the rich template gallery that can be used as a starting point. The gallery contains all the major open source implementations for CMS, Blogging, eCommerce, Galleries and Forum websites.

Examples include: Joomla, Umbraco, MojoPortal, nopCommerce Drupal, Moodle, WordPress, DasBlog and many more.

Go to http://www.microsoft.com/web/webmatrix/ and download WebMatrix.

2. Start WebMatrix

image

3. Create your Orchard WebSite

    • Click “New” –> “App Gallery”
    • Select Orchard CMS and click “Next”

image

  • Click next again
  • Accept the EULA
  • Wait until the download completes.

When the download completes, Orchard will start and open an initial configuration page in the browser:

image

  • Fill in the Site Name, an admin username and password
  • Select your data storage: SQL Server Compact, SQL Server or MySql
  • Select an Orchard Recipe. If you want to create a blog, select blog, if not, select default.
  • Click Finish Setup

And you are done, your website is running on your local machine:

image

From the home website you can access the dashboard via the link at the bottom. The dashboard contains all the configuration options for settings, content, etc.

image

 

In the next post we will explore the features and modules that you need after you have completed the orchard installation:

http://www.12qw.ch/2014/03/getting-started-with-orchard-cms-the-modules-and-features-you-need

Category: Business  Tags: , ,  Leave a Comment

Azure Friday – Get started with the Azure Cloud!

Do you want to get started with Windows Azure Virtual Machines?

The easiest way to learn and get to know the Azure VM features is Azure Friday. A collection of bite-size (10-15mins) videos on Windows Azure hosted by Technologist Scott Hanselman and all the Azure guys such as Scott Guthrie and others.

Check it out today!

http://www.windowsazure.com/en-us/documentation/videos/windows-azure-friday/

Category: Business  Tags:  Leave a Comment

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 7: The Sample Application 3

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

Writing the StreamInsight Queries

The sample code to the application can be downloaded here:

Scenario 1: Tweets per Second

As we can see from our pass-through implementation from the last post, we receive a large amount of tweets. In the first query we would like to find out how many tweets we receive per second. This turns out to be pretty easy!

image

We define a tumbling window with a length of one second and just call the count property. StreamInsight uses three different types of windows:

  • TumblingWindow: A fixed length window. The next window begins when the current one ends.
  • HoppingWindow: A fixed length overlapping window. We can define the time interval in which we want the next window. Example: We have a 3 second window that moves in one second hops. So we get updated data every second for the last 3 seconds.
  • SlidingWindow: The sliding window reacts to the input stream and always moves when there is a change in the stream. The window size is defined by two adjacent events.

In order for the example to work, we have to do two more things. First, we need to adjust the consoleObserver. Since the query produces scalar count values (and not TweetItems anymore) we need to adjust the generic type parameter in the DefineObserver() method from TweetItem to long.

image

Next, we need to replace the twitterstream instance with the query for the Bind() call. We now have a query that operates on the data source and bind the observer to the query.

image

That’s it! The output should look somewhat like this:

image

Amazing stuff already, but wait, there’s more!

Tweets per Second grouped by Language

For the next query we are going to group the tweets by language. We first need a data object for the computed output. As a little twist we add a method call that resolves the culture info that we find in the Tweet.

image

Then we continue and write our query. Remember that we have to adjust the generic type parameter of the observer to use the new LanguageSummary type:

image

The query leads to output like this:

image

Nice.

Language with most Tweets per Second

Next, we improve the last query by adding another query that uses the output of the last example as input. We want to know the top 5 languages in terms of tweets per second. We use the SnapShot window here to react in changes in the stream, that is changes in the output of the last query.

image

image

The most popular Tweet every Second

Next, we want to find the most popular Tweet every three seconds. First, we create a new data class:

image

Then we write a query that groups all the tweets in every 3 second window by the number of followers that the user has:

image

image

Note: The numbers after the user name show (Followers/Friends).

In the next step we improve the last example by adding the Friendcount. We want to know the person in every three second window that has most followers AND most friends. We implement this by writing two different queries for followers and friends and then join them together using a StreamInsight join operation.

Let’s start by writing the most friend query first. It is similar to the followers query:

image

Now we use the join to join the queries on the user name:

image

And that is it. If we run the sample we find out who the most popular person on twitter is ever three seconds:

image

An interesting observation that we can make looking at the data is that in some windows, e.g. 06:21:57, there is no event. This means that at that point the person with most tweets and most friends was not the same person.

The join is a very powerful operation that can give us deep insight into the data.

If you want to play around and try out more complex queries, look at the resources in the first blog post of the series, especially at the LINQPad Samples and the Hitchhiker guide.

I hope you enjoyed this series on Microsoft StreamInsight. If it provided substantial value to you, feel free to donate. :-)

Category: Business  Tags: , , ,  2 Comments

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 6: The Sample Application 2

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

The sample code to the application can be downloaded here:

Putting it all together

Now that we have written our TweetItem, TwitterStream an Unsubscriber class, we can put it all together and write the actual StreamInsight code. We start by implementing the main method.

First we define the StreamInsight Server. In the Create() method we specify the name of the StreamInsight instance as we defined it during the installation. Then we create a StreamInsight application that will hold our data sources, sinks and queries.

image

Next, we define our data sink. We write a method that prints the event to the console. There is an IF statement that only prints the stream events that we are interested in and omits the CTI’s. We implement it as a generic method with a type variable for the payload. This allow us to reuse it when we have different queries with different output POCOs:

image

Following the StreamInsight 2.1 approach we can create a sink around the method above and hook it to the application in one line. Note: You have to include the Microsoft.ComplexEventProcessing.Linq namespace in order to see the DefineObserver() extension method.

image

However, it is one line but to my opinion it is hard to understand. Let’s pick it apart and analyze what is going on:

image

We use the static Create() method from the observer class to create an Observer instance that can observer PointEvent<object> and we point it to the ConsoleWritePointNoCTI method that accepts an element of type PointEvent<object> as a parameter.

Next, we create our data source from the TwitterStream.cs class.

image

Here as well, we can write it in one line but it is even more complicated than the last statement. Lets look at what is going on:

image

We can see in the code that we begin by creating an instance of our TwitterStream class. Remember that this is the class where we implemented the IObservable<TweetItem> interface.

Then we use the DefineObservable() extension method from the StreamInsight assembly to convert the IObservable into an IQbservable. From there, we convert the IQbservable into an IQStreamable. In this call we define the structure of our PointEvent items and instruct StreamInsight to add CTIs to the stream.

Finally, we create a binding that connects the data source and the data sink and we call the Run() method on that binding. In this first example, we do not yet use a query and just run all the data that we receive in the source directly to the sink.

image

And that’s it! Hit F5 and HERE WE GO!

The output should look something like this:

image

Note: The question marks come from languages with characters that can not be printed by the standard console.

Bravo! You can now receive live Twitter data processed by StreamInsight!

Next week, we will write our first queries and process the data between the source and the sink.

Category: Business  Tags: , , ,  2 Comments

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 5: The Sample Application 1

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

The sample code to the application can be downloaded here:

Writing the Sample Application

In this post we are going to write a sample application from scratch. We start by installing StreamInsight and then jump into Visual Studio and start coding the project. First, we are going to use a library to get access to the Twitter feed. As soon as we have the basics we elaborate on some interesting queries on the Twitter data stream.

Installing StreamInsight

First, we need to download the StreamInsight installer executable. We can find it in the Microsoft Download Center under: http://www.microsoft.com/en-us/download/details.aspx?id=30149

We need the StreamInsight.msi Installers corresponding to your target platform bitness.

During the installation you will be asked if you would like to create a StreamInsight instance. Do so and give it any name you like. In this example I am going to name it “Thomas”.

Creating the Visual Studio Solution

Now, let’s fire up Visual Studio. I am using VS2012 but you can use VS2010 as well. We make it simple and create a new console project. Then we need to add the Microsoft StreamInsight assemblies. Add assembly references to the following files:

  • Microsoft.ComplexEventProcessing: Found in the StreamInsight install directory: C:\Program Files\Microsoft StreamInsight 2.1\Bin\
  • System.Reactive: Found under .NET 4.0
  • System.Reactive.Providers: Found under .NET 4.0

Your References should look like this:

image

Next, we integrate Twitter into our solution. We use a handy library named Tweetinvi and install it via NuGet Package Manager. Maybe you need to install the NuGet Package Manager in your Visual Studio, if it is not available. Right-click on the References folder in the Solution Explorer and select “Manage NuGet Packages”. Do an online search for “Tweetinvi” and hit “install”.

image

First we are going to write a data class that will be our payload object. We call it TweetItem and fill it with a couple of properties. For the sake of simplicity, we restrict our item to the following fields: User, Text, CreationDate, Language, Followers, Friends.

image

In our TweetItem class we provide a constructor that accepts an object that implements the ITweet interface and converts it into a TweetItem. The ITweet interface can be found in the Tweetinvi Library and contains all the fields that we get for each tweet.

 

Creating the Twitter Observable

Next, we create the class that we use as our data source and feed into StreamInsight. Create a new class called TwitterStream.cs and implement the IObservable<TweetItem> interface. The interface contains a Subscribe() method:

image

First, we are going to add a list of IObserver<TweetItem> objects. Then we add code to the Subscribe() method that adds the observer parameter to the observer list.

image

The result of the subscribe method is an object that implements IDisposable. The subscriber can use it to cancel his subscription. The subscription will end when the subscriber disposes the object that he received upon subscription. We implement a small Unsubscriber class that implements IDisposable and keeps references to the observer list and its own observer instance. Upon disposal, it removes its own reference from the observer collection.

image

Next we need a Start() method that connects to twitter and starts receiving Tweets. In order to receive Tweets from Twitter you will need to register on the Twitter website and get the following four credential keys: UserKey, UserSecret, ConsumerKey, ConsumerSecret. You can start on: https://dev.twitter.com/docs. Once you can login to dev.twitter.com you can create an application and will get your authentication information.

There is a free subscription to Twitter that will give you one percent of all the worldwide Tweets. Once you have your credentials, we can implement the start method:

image

The start method uses the Token and SimpleStream classes from the Tweetinvi namespace. Note that the SimpleStream class has nothing to do with StreamInsight. In the StartStream() method we provide a lambda expression that points to the OnNewTweet(tweet) method that we will define next. The OnNewTweet() method is used to map the Tweets into TweetItems and forward each Tweet that we receive to all the items in our observer collection:

image

Finally, we implement the constructor. We initialize the myObservers collection and we call the Start() method. We execute the Start() method on a background thread using the Task Parallel Library (TPL).

image

In the next post we put it all together and write our first query. Stay tuned!

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 4: Reactive Programming

Reactive Programming with the Rx Extensions

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

In this chapter we are going to look at the StreamInsight 2.1 approach. We analyze and try to understand the interfaces that we met at the end of the last post.

image

We start by looking at where the different interfaces have their origin. This will give us clues on the functionality that they provide:

image

IEnumerable<T>: The IEnumerable interface comes from .NET 2.0 and has been there forever. We use it to describe an enumerable collection of items. All the common collections in .NET, such as Lists, Arrays, Sets, etc. implement the IEnumerable interface. It enriches the collection with functionality and is required if you want to “foreach” over a collection.

IQueryable<T>: IQueryable is the dual to IEnumerable. It serves the same purpose, describing a collection that contains enumerable items. The difference lies in the implementation. IQueryable can be used to access a remote collection of items. The interface contains an abstraction of the functionality of IEnumerable that can be projected onto any remote system, if there is a matching query provider available.

IObservable<T>: The IObservable interface can be found in the .NET 4.0 namespace. It fits together with the IObserver<T> interface and can be used to push data from the observable object to the observer object. We use it to implement publish-subscribe.

IQbservable<T>: Probably the interface with the strangest name ever, right? By the way, the Q stands for “Query”, as in IQueryable. IQbservable is to IObservable what IQueryable is to IEnumerable. This means that we can use IQbservable to implement the push paradigm with an observer that resides on a remote system.

IQStreamable<T>: IQStreamable is the interface that StreamInsight uses to represent a data stream.

Let’s look at the interfaces from a functional viewpoint:

image

On this image we can see how the interfaces are related in terms of PULL vs. PUSH and LOCAL vs. REMOTE.

Lets take a closer look at the difference in implementation of the PULL and the PUSH versions of the interfaces.

image

We can see that IEnumerable defines a GetEnumerator() method. The enumerator contains methods to navigate through the items in the collection.

On the other hand, the IQueryable interface is much more abstract. It provides an expression and a queryprovider. The expression is a statement represented in the form of an expression tree. An expression  tree is an abstract construct that can represent a query such as the where clause in a LINQ statement.

The idea is that due to the abstract implementation of the expression, it can be sent to a remote system. On the remote system. The query provider is responsible for translating the expression into the language of the remote system. If we think about SqlServer, this might be SQL syntax.

We can see that the relation between IObservable and IQbservable has exactly the same purpose:

image

The IObservable interface provides an OnNext() method. Recall that we use it for publish-subscribe purposes. The OnNext() method is the place where the Observable pushes the data to the Observer. The IQbservable represents the logic in an expression and a query provider for remote execution.

WPF Samples: Real World WPF Line-of-Business (LOB) Applications

Have you ever wondered what WPF LOB Applications are out there in the wild?

Pete Brown published an interesting collection of real world WPF applications called the Book of WPF. It is already a couple of years old but still interesting to look at:

http://10rem.net/blog/2010/02/10/the-book-of-wpf

You can download the PDF directly here:

http://archive.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=WPFPresents&DownloadId=8223

Somehow I got my hands on the following Power Point Deck. I could not find it anymore. It contains some of the projects from the Book of WPF in a more shiny way:

WPF

Very interesting stuff:

image

image

image

image

image

Category: Business  Tags: , ,  Leave a Comment

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 3: StreamInsight Programming

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

In this part, we ar going to look at the Streaminsight programming model. 

What do we need?

  • Visual Studio. In order to use StreamInsight there is always some .NET coding required.
  • A hosted StreamInsight server. The server can be installed from the StreamInsight executable as a windows service. (about 10MB)
  • The StreamInsight Installer executable. It contains the server component and the assemblies that need to be referenced in Visual Studio.
  • An SqlServer license. For productive use of StreamInsight, we need a valid SqlServer license.

Important to note:

  • We do not need an SqlServer instance in order to use StreamInsight
  • StreamInsight is more a framework than a complete product. There is always some coding involved.

StreamInsight Artifacts

This section will explain the artifacts and concepts that need to be understood in order to start developing the StreamInsight solution.

image

A StreamInsight solution always consists of the items listed below. In the demo application we will see all of these again in our code.

  • A StreamInsight server. The server usually runs as a windows service and can provide multiple instances where we deploy our data sources, sinks and queries.
  • A StreamInsight Application. The StreamInsight application is a logical container in the server that contains our artifacts.
  • A data source. The source can either be standing, e.g. historical data or it can be moving data from a live data stream. implementation is the same.
  • A data sink. The place where StreamInsight puts its output.
  • A query. The query is written in LINQ
  • A binding. The binding connects the query to a sink.

StreamInsight Event Types

The data items that we feed from our .NET code into and out of the StreamInsight engine are always generic types with a POCO payload object. These generic types contain temporal information such as start and end times plus the POCO payload object. We differentiate three different types according to the nature of the event:

  • PointEvent<T>: A point event is an event with a single timestamp. It can be used for events that occur exactly once and would be visualized in the form of a point.
  • IntervalEvent<T>: An interval event is an event with a start time, an end time and a fixed duration larger than 0.
  • EdgeEvent<T>: Edge events are as well defined by a start and an end point. The difference to IntervalEvent is that as the start time passes and the interval begins, we do not yet know either the duration nor the end time of the event.

Another event type in contrast to the data events are Current Time Increments (CTIs). StreamInsight needs to have CTI events in the stream that it is going to analyze. The developer has to define how CTI are inserted into the stream. CTIs mark constant time windows inside the stream:

image

StreamInsight needs CTIs to process the stream for the following reasons:

  • Stream processing: A CTI “closes” the current timeframe. Let’s make an example: If we work with 1-minute CTIs, when the 12:05 CTI arrives, StreamInsight will process the 12:04 – 12:05 window. If we have late arriving events that belong to the window but arrive after the 12:05 CTI, StreamInsight will ignore them.
  • Stream sync: If we evaluate data from multiple streams, StreamInsight will synchronize the two streams based on their respective CTIs.
  • Non-events: We can look for missing events in CTI windows. We could use this approach to implement a monitoring solution that regularly checks on a heartbeat signal.

 

Getting the data in and out of StreamInsight

In order to connect StreamInsight to data sources or data consumers we need to start programming.

StreamInsight knows 2 different implementation approaches. The first one is the adapter approach. It has been there forever and can be used for complex input and output. As of StreamInsight 2.1, the adapter approach is often called the legacy approach.

The second approach is a feature of StreamInsight 2.1 and relies on existing classes from .NET or from the .NET Reactive Extensions (Rx Extensions). The idea is that if we already have a class that implements the interfaces IEnumerable<T> or IObservable<T> we can hook these into our processing.

 

The Adapter (Legacy) Approach

The adapter approach is pretty straightforward. We use the IInputAdapter<T> and IOutputAdapter<T> interfaces and implement them in our classes. Then we provide a config and a factory class and instruct StreamInsight to request an adapter from the factory.

image

 

The StreamInsight 2.1 Interface Approach

The interface approach is said to be much easier. I personally think that it is way harder to understand for a beginner. There are a lot of interfaces and once you grasp the purpose of each one, it starts to get more understandable.

image

We have three groups of interfaces that we use depending on the nature of our data source:

  • IEnumerable<T>/IQueryable<T>: If we have a collection as a data source.
  • IObservable<T>, IQbServable<T>: If we have a stream as a data source.
  • IQStreamable<T>: IQStreamable is the interface that StreamInsight understands and describes a data stream.

StreamInsight provides extension methods that let us create an IQStreamable object from any of the other interfaces. This is very convenient for us. In short, this means that we have to create or convert our data source into IEnumerable (for PULL) of IObservable (for PUSH) and can then convert this object in to an IQStreamable.

I will go into the details of all the interfaces in the next chapter.

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 2: What are CEP and StreamInsight?

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

What is CEP?

Wikipedia defines Event Processing as:

“Event processing is a method of tracking and analyzing (processing) streams of information (data) about things that happen (events), and deriving a conclusion from them”

and Complex Event Processing as:

“event processing that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) and respond to them as quickly as possible.”
http://en.wikipedia.org/wiki/Complex_Event_Processing

CEP is often used for the following types of applications:

  • algorithmic trading
  • pattern detection (e.g. fraud detection)
  • security applications
  • sensor data, e.g. in robotics

Google-Car
Example for a CEP scenario: The sensor data of the Google car.

Data-Visualzation
Example for a CEP scenario: Processing and visualization of financial market data

CEP Solutions

The following image contains an overview of a typical CEP architecture. We use data from devices, sensors, computers, data stores or other data sources and feed them, often in real time, into a CEP engine. The engine uses standing queries to analyze the data streams on the fly. The output of the query is forwarded to targets that typically alert, visualize, store or forward the data.

image

Two important points to note:

  • In CEP we are often more interested in events (and non-events) or aggregates of data rather than the data itself
  • StreamInsight must not be confused with HDInsight. HDInsight is the Big Data solution (Hadoop) offered on Windows Azure.

What is Microsoft StreamInsight

StreamInsight is a CEP Engine and framework developed by Microsoft. It belongs to the Sql Server stack but it can be used totally independent of SqlServer.

However, in order to use StreamInsight on a productive system you need an SqlServer License. In order to get to know the product there is a free 180-day trial license available. (Can be selected during installation).

I am a huge StreamInsight fan since it is very lightweight, extremely powerful and to me as a .NET developer it is really easy to use. The query system in StreamInsight extends the Language Integrated Query (LINQ) concept that should be familiar to every .NET developer.

I highly recommend the following TechNet article to get a nice overview about StreamInsight:

StreamInsight for non-programmers:
http://social.technet.microsoft.com/wiki/contents/articles/14437.streaminsight-for-non-programmers.aspx

If we look at the architecture of a StreamInsight solution, it is totally coherent to the CEP architecture depicted above:

image

In StreamInsight we speak of data sources, the objects that feed the data into StreamInsight and data sinks, the objects that receive a data feed from StreamInsight. Note that the StreamInsight query language is LINQ and we even have the possibility to combine multiple queries inside the engine.

In the next post we are going to take a closer look at the StreamInsight programming model: 

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 3: StreamInsight Programming

 

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 1: Introduction and Resources

Introduction and Resources

Welcome to my 7-post series on Complex Event Processing with Mircosoft StreamInsight. Enjoy!

Microsoft StreamInsight is a very lightweight, powerful and for .NET developers easy to use Complex Event Processing (CEP) engine. It uses .NET LINQ queries to write queries on data streams.

With minimal effort, StreamInsight allows you to answer questions such as “what is the one minute aggregate of my value X grouped by dimension D”. This series of blog posts explains the concepts behind CEP and StreamInsight and implements a sample application that can be used to analyze live data from Twitter.

Index of this blog series:

Part 1 (03.10.13): CEP with StreamInsight and Twitter 1:
Introduction and Resources

Part 2 (06.10.13): CEP with StreamInsight and Twitter 2:
What are CEP and StreamInsight

Part 3 (13.10.13): CEP with StreamInsight and Twitter 3:
StreamInsight Programming

Part 4 (20.10.13): CEP with StreamInsight and Twitter 4:
Reactive Programming

Part 5 (27.10.13): CEP with StreamInsight and Twitter 5:
The Sample Application 1

Part 6 (03.11.13): CEP with StreamInsight and Twitter 6:
The Sample Application 2

Part 7 (10.11.13): CEP with StreamInsight and Twitter 7:
The Sample Application 3

The best resources that I found:

Video: “What’s new in Complex Event Processing using StreamInsight for SqlServer 2.1” by Torsten Grabs, Principal Program Manager Lead, Microsoft

http://www.youtube.com/watch?v=2k1AO5Wr_kk

Video: “My Campus Walk: I walk, StreamInsight analyzes, Worldwide Telescope visualizes.” Interesting project about visualization of a walk path from geo-spatial data processed by StreamInsight.

http://www.youtube.com/watch?v=Zi7JJmC4Nxw

Video: “Heat mapping H5N1 Outbreaks with StreamInsight and Binq Maps”. Really awesome demo application that feeds processed stream data to a Bing Maps application.

http://www.youtube.com/watch?v=cctmJZZDBlg

Tools

A tool I absolutely recommend, especially for StreamInsight Development,  is LINQPad (http://www.linqpad.net/). LINQPad includes the following StreamInsight sample packages that can be executed and played with:

      • 101 StreamInsight Samples
      • Hitchhikers Guide to StreamInsight
      • Performance Counter Samples
      • Reactive Extensions
      • LINQ in Action

Important Note:
If you use the free version of LINQPad you will not get any intellisense or auto-complete functionality. In my opinion this makes it useless. As soon as I bought a personal license (about USD30), LINQPad turned into one of my favorite dev tools of all times. It is extremely lightweight and understands C#, VB, F#, SQL, LINQ, ADO.NET, StreamInsight, etc.

Samples

TwitterStream Sample: The application that we are going to develop in the following blog posts:

 

Hitchhikers Guide to StreamInsight Queries: Very comprehensive guide to writing queries in StreamInsight. Code is included in the codeplex download below. Samples are included in LINQPad.
http://blogs.msdn.com/b/streaminsight/archive/2012/08/01/a-hitchhiker-s-guide-to-streaminsight-2-1-queries.aspx

StreamInsight Product Team Samples 2.1
http://streaminsight.codeplex.com/releases/view/90143

StreamInsight Product Team Samples 2.0
Very important collection of samples such as: TextFileReaderAdapter, TextFileWriterAdapter, SqlAdapter, WcfAdapter and lots of applications.
http://streaminsight.codeplex.com/releases/view/46435

JAhlen StreamInsight Example Adapters v2.1
Stock trading sample application.
http://streaminsight.codeplex.com/releases/view/97583

 

Blogs/MSDN/Technet/Forums:

Blog: Official Blog of the Microsoft StreamInsight Dev Team
http://blogs.msdn.com/b/streaminsight/

Blog: StreamInsight explained to non-programmers
http://social.technet.microsoft.com/wiki/contents/articles/14437.streaminsight-for-non-programmers.aspx

Blog: Getting Started with StreamInsight 2.1
http://blogs.msdn.com/b/streaminsight/archive/2012/09/18/getting-started-with-streaminsight-2-1.aspx

Technet: Microsoft StreamInsight
http://technet.microsoft.com/en-us/library/hh750618(v=sql.10).aspx

MSDN: Microsoft StreamInsight
http://msdn.microsoft.com/en-us/library/hh750618(v=sql.10).aspx

MSDN Forum: SQL Server –> Microsoft StreamInsight
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/home?forum=streaminsight

 

Next Post:

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 2: What are CEP and StreamInsight?