Performance and Diagnostics Hub

Performance Profiling of .NET Applications in Visual Studio 2013/2015

There have been a lot of improvements in the troubleshooting and especially profiling capabilities of Visual Studio over the last years.

While in VS2010 the Visual Studio Profiler was restricted to owners of the premium and ultimate versions, the Visual Studio team made a clever move in shifting the profiler down into the professional versions of VS2012. So profiling .NET applications by use of sampling and instrumentation became a tool for everybody. However, this act of kindness by Microsoft got mostly ignored by the community.  Many developers still do not realize that they have a powerful tool in-the-box on their box!

Behold, it gets even better. With the release of Visual Studio 2013, the Performance & Diagnostics Hub was introduced. A central platform for everything you need related to application diagnostics and profiling.

The hub in VS2013 is under:

ANALYZE –> PERFORMANCE AND DIAGNOSTICS (Alt-F2)

or in VS2015:

ANALYZE –> START DIAGNOSTIC TOOLS WITHOUT DEBUGGING (Alt-F2)

Now that you know where it is, GO AND USE IT!!!

The idea of the hub is to unite several simple diagnostic tools. The current list in VS2013 Update 4 is:

  • CPU Usage
  • GPU Usage
  • HTML UI Responsiveness(Store/Phone Apps Only)
  • XAML UI Responsiveness (Store/Phone Apps Only)
  • Memory Usage
  • JavaScript Function Timing (Store/Phone Apps Only)
  • JavaScript Memory (Store/Phone Apps Only)
  • Performance Wizard (a.k.a Visual Studio Profiler)
  • Energy Consumption (Store/Phone Apps Only)

As you can see, most of the new tools can only be used with store apps. However, Microsoft aims to expand the reach of the tools to other application types, such as WPF or ASP.NET.  Updates are coming in quickly with VS updates.

Therefore, in Visual Studio 2015, the XAML UI Responsiveness Tool has been renamed and now fully supports Windows Presentation Foundation( WPF)!

On another note, the Performance & Diagnostics Hub is now available in all the VS editions, which includes the new VS Community Edition.

vs2013community

The tools in the hub mostly rely on Event Tracing for Windows (ETW). They record ETW events, collect data from other sources, correlate and create graphic reports for the developer to analyze.

Stay tuned for more posts on performance profiling in Visual Studio! In the meantime, grab the current edition of the Windows Developer magazine and read my article on the profiling tools:

http://windowsdeveloper.de/artikel/Was-laeuft-hier-eigentlich-178995

Metro Studio

Free XAML Icons and Shapes in Metro Studio

In any UI Framework, visuals are the key to success. An application that contains appealing graphics automatically yields a higher user acceptance.

Unfortuantely, creating good icons is usually very hard and expensive. Not anymore! Meet Syncfusions Metro Studio!

The Syncfusion Metro Studio is a collection and editor for over 4000 application icons that can be exported to XAML! And the best part is: IT IS FREE and the icons can be used in commercial applications.

Get your copy now at:

http://www.syncfusion.com/downloads/metrostudio

With a few clicks you can create icons such as these:

image

The editor allows you to change colors, shape, size etc.

Once your icon is finalized, it can be exported to XAML and integrated into your XAML application. In addition to XAML, the icons can be exported to PNG, GIF, JPG, BMP, ICO,  TIFF or SVG.

As far as I know, this is by far the best and most affordable icon library that there is. A huge time-saver!

base64
1 Comment

ToolTip: Decoding Base64 Images with Chrome Data URL

Reading the great blog post about Base64 encoding and decoding in .NET/C# written by Jerry Nixon (http://blog.jerrynixon.com/2014/11/reading-and-writing-base64-in-windows.html), I immediately remembered a great trick that comes in very handy when using Base64 encoding.

Suppose you have an image that you would like to encode into base64:

sn

You will end up with a very long string, such as this one:

/9j/4AAQSkZJRgABAQEAYABgAAD/4QBKRXhpZgAATU0AKgAAAAgAAwEaAAUAAAABAAAA MgEbAAUAAAABAAAAOgEoAAMAAAABAAIAAAAAAAA5OHAAAJiWgDk4cAAAmJaA/+EF7Gh 0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8APD94cGFja2V0IGJlZ2luPSfvu78nIGlkPSd XNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQnPz4NCjx4OnhtcG1ldGEgeG1sbnM6eD0iYW RvYmU6bnM6bWV0YS8iPjxyZGY6UkRGIHhtbG…

During testing of your application, it can be really painful to convert these strings back into jpg files just to see if nothing got missing or corrupted. You could either write a small program, that decodes the base64 string, use a tool or you could use an online base64 decoder/encoder such as this one:

http://www.freeformatter.com/base64-encoder.html

Aaaand, here comes the trick:

If you have Google Chrome installed, you can use a feature called Data Uri to decode your Base64 string. Just type into the chrome address bar:

data:image/jpeg;base64,<base64>

where <base64> is the base64 encoded data. In our example:

data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/4QBKRXhpZU0…

And BOOM, chrome decodes the Base64 data and displays our image!

image

You can even specify a different target type, such as:

data:image/png;base64,<base64>

or

data:image/webp;base64,<base64>

I like this trick very much. It is a huge timesaver! I hope you like it too!

image.png

Microsoft TechTalk: S.O.L.I.D. Principles and DDD

On September 18, 2014, I held a Tech Talk at the Microsoft Switzerland Offices in Walisellen. The title was “Enterprise Software Architecture and Domain-Driven Design”. The main goal of the talk was to explain that with the .NET Framework getting more and more mature, it becomes ever more important to think about how we deal with legacy .NET code, that is existing .NET code.

Everyday, more and more .NET projects will not start from scratch but start on an already existing code base. Therefore, we as developers need to understand how to reduce our legacy by writing more maintainable code.

The SOLID Principles are an approach on how to write modular code that enables reuse scenarios. Robert C. Martin, aka. Uncle Bob first came up with the principles in the early 2000s.

image

The SOLID principles are:

  • SRP: Single Responsibility Principle
  • OCP: Open/Closed Principle
  • LSP: Liskov Substitution Principle
  • ISP: Interface Segregation Principle
  • DIP: Dependency Inversion Principle

Each of the principles describes an approach to “correctly do object oriented programming (OOP)”.

After I explained the SOLID Principles and showed a couple of demos, we took a look at Domain Driven Design (DDD). DDD is an approach to designing a software solution by following a couple of recommendations.

image

It concentrates on creating a common understanding between the business experts and the developers by creating artifacts such as a common language (the ubiquitous language) and establishing a common model (the domain model), which is a graphical representation of the business domain. Using several DDD techniques, the model is iteratively refined by making it more correct and reducing complexity. A very important concept is to make implicit stuff explicit. Often things that seem simple or even trivial to the developer are absolutely crucial to the business. Therefore, it is vital to give these concepts a name and add them to the model as a visible artifact.

image

Using further techniques such as entities, aggregates, value objects, factories or repositories, the model is then translated into code.

Attached is the slide deck that I used during the presentation.

A big thank you goes out to Microsoft Switzerland for giving me the opportunity and to the engaged crowd for listening and thinking along!

2 Comments

WPF: The Simplest Way to Get the Default Template of a Control as XAML

Every WPF Developer has probably faced this issue before. You want to change the default template of a control and need it in XAML. Where can you get the default template? There are multiple ways:

  1. Find the correct MSDN page for your framework version
  2. Use a tool
  3. Use Expression Blend

In my opinion, none of these solutions is very convenient. I always end up either not finding the correct version, not having blend installed, etc…

Alas, there is a much easier way. You can use the XamlWriter class to serialize the default template of your control:

1 public static void SaveDefaultTemplate() 2 { 3 var control = Application.Current.FindResource(typeof(ProgressBar)); 4 using (XmlTextWriter writer = new XmlTextWriter(@"defaultTemplate.xml", System.Text.Encoding.UTF8)) 5 { 6 writer.Formatting = Formatting.Indented; 7 XamlWriter.Save(control, writer); 8 } 9 }

This will produce the following file for the ProgressBar in the sample code:

1 <Style TargetType="Button" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib"> 2 <Style.BasedOn> 3 <Style TargetType="ButtonBase"> 4 <Style.Resources> 5 <ResourceDictionary /> 6 </Style.Resources> 7 <Setter Property="FrameworkElement.FocusVisualStyle"> 8 <Setter.Value> 9 <Style TargetType="IFrameworkInputElement"> 10 <Style.Resources> 11 <ResourceDictionary /> 12 </Style.Resources> 13 <Setter Property="Control.Template"> 14 <Setter.Value> 15 <ControlTemplate> 16 <Rectangle Stroke="{DynamicResource {x:Static SystemColors.ControlTextBrushKey}}" StrokeThickness="1" StrokeDashArray="1 2" Margin="2,2,2,2" SnapsToDevicePixels="True" /> 17 </ControlTemplate> 18 </Setter.Value> 19 </Setter> 20 </Style> 21 </Setter.Value> 22 </Setter> 23 <Setter Property="Panel.Background"> 24 <Setter.Value> 25 <SolidColorBrush>#FFDDDDDD</SolidColorBrush> 26 </Setter.Value> 27 </Setter> 28 <Setter Property="Border.BorderBrush"> 29 <Setter.Value> 30 <SolidColorBrush>#FF707070</SolidColorBrush> 31 </Setter.Value> 32 </Setter> 33 <Setter Property="TextElement.Foreground"> 34 <Setter.Value> 35 <DynamicResource ResourceKey="{x:Static SystemColors.ControlTextBrushKey}" /> 36 </Setter.Value> 37 </Setter> 38 <Setter Property="Border.BorderThickness"> 39 <Setter.Value> 40 <Thickness>1,1,1,1</Thickness> 41 </Setter.Value> 42 </Setter> 43 <Setter Property="Control.HorizontalContentAlignment"> 44 <Setter.Value> 45 <x:Static Member="HorizontalAlignment.Center" /> 46 </Setter.Value> 47 </Setter> 48 <Setter Property="Control.VerticalContentAlignment"> 49 <Setter.Value> 50 <x:Static Member="VerticalAlignment.Center" /> 51 </Setter.Value> 52 </Setter> 53 <Setter Property="Control.Padding"> 54 <Setter.Value> 55 <Thickness>1,1,1,1</Thickness> 56 </Setter.Value> 57 </Setter> 58 <Setter Property="Control.Template"> 59 <Setter.Value> 60 <ControlTemplate TargetType="ButtonBase"> 61 <Border BorderThickness="{TemplateBinding Border.BorderThickness}" BorderBrush="{TemplateBinding Border.BorderBrush}" Background="{TemplateBinding Panel.Background}" Name="border" SnapsToDevicePixels="True"> 62 <ContentPresenter RecognizesAccessKey="True" Content="{TemplateBinding ContentControl.Content}" ContentTemplate="{TemplateBinding ContentControl.ContentTemplate}" ContentStringFormat="{TemplateBinding ContentControl.ContentStringFormat}" Name="contentPresenter" Margin="{TemplateBinding Control.Padding}" HorizontalAlignment="{TemplateBinding Control.HorizontalContentAlignment}" VerticalAlignment="{TemplateBinding Control.VerticalContentAlignment}" SnapsToDevicePixels="{TemplateBinding UIElement.SnapsToDevicePixels}" Focusable="False" /> 63 </Border> 64 <ControlTemplate.Triggers> 65 <Trigger Property="Button.IsDefaulted"> 66 <Setter Property="Border.BorderBrush" TargetName="border"> 67 <Setter.Value> 68 <DynamicResource ResourceKey="{x:Static SystemColors.HighlightBrushKey}" /> 69 </Setter.Value> 70 </Setter> 71 <Trigger.Value> 72 <s:Boolean>True</s:Boolean> 73 </Trigger.Value> 74 </Trigger> 75 <Trigger Property="UIElement.IsMouseOver"> 76 <Setter Property="Panel.Background" TargetName="border"> 77 <Setter.Value> 78 <SolidColorBrush>#FFBEE6FD</SolidColorBrush> 79 </Setter.Value> 80 </Setter> 81 <Setter Property="Border.BorderBrush" TargetName="border"> 82 <Setter.Value> 83 <SolidColorBrush>#FF3C7FB1</SolidColorBrush> 84 </Setter.Value> 85 </Setter> 86 <Trigger.Value> 87 <s:Boolean>True</s:Boolean> 88 </Trigger.Value> 89 </Trigger> 90 <Trigger Property="ButtonBase.IsPressed"> 91 <Setter Property="Panel.Background" TargetName="border"> 92 <Setter.Value> 93 <SolidColorBrush>#FFC4E5F6</SolidColorBrush> 94 </Setter.Value> 95 </Setter> 96 <Setter Property="Border.BorderBrush" TargetName="border"> 97 <Setter.Value> 98 <SolidColorBrush>#FF2C628B</SolidColorBrush> 99 </Setter.Value> 100 </Setter> 101 <Trigger.Value> 102 <s:Boolean>True</s:Boolean> 103 </Trigger.Value> 104 </Trigger> 105 <Trigger Property="ToggleButton.IsChecked"> 106 <Setter Property="Panel.Background" TargetName="border"> 107 <Setter.Value> 108 <SolidColorBrush>#FFBCDDEE</SolidColorBrush> 109 </Setter.Value> 110 </Setter> 111 <Setter Property="Border.BorderBrush" TargetName="border"> 112 <Setter.Value> 113 <SolidColorBrush>#FF245A83</SolidColorBrush> 114 </Setter.Value> 115 </Setter> 116 <Trigger.Value> 117 <s:Boolean>True</s:Boolean> 118 </Trigger.Value> 119 </Trigger> 120 <Trigger Property="UIElement.IsEnabled"> 121 <Setter Property="Panel.Background" TargetName="border"> 122 <Setter.Value> 123 <SolidColorBrush>#FFF4F4F4</SolidColorBrush> 124 </Setter.Value> 125 </Setter> 126 <Setter Property="Border.BorderBrush" TargetName="border"> 127 <Setter.Value> 128 <SolidColorBrush>#FFADB2B5</SolidColorBrush> 129 </Setter.Value> 130 </Setter> 131 <Setter Property="TextElement.Foreground" TargetName="contentPresenter"> 132 <Setter.Value> 133 <SolidColorBrush>#FF838383</SolidColorBrush> 134 </Setter.Value> 135 </Setter> 136 <Trigger.Value> 137 <s:Boolean>False</s:Boolean> 138 </Trigger.Value> 139 </Trigger> 140 </ControlTemplate.Triggers> 141 </ControlTemplate> 142 </Setter.Value> 143 </Setter> 144 </Style> 145 </Style.BasedOn> 146 <Style.Resources> 147 <ResourceDictionary /> 148 </Style.Resources> 149 </Style>

Sun And Clouds Wallpapers (1)

AzureAdventures: Setting up the Adventureworks Database in under 10 Minutes

This blog post will show you how easy it is to set up the famous Adventureworks database using the Microsoft Azure Cloud.

Before we start, a little background:

A couple of days ago, I was at a customer site and needed a simple database in order to show a .NET framework demo. I wanted to demonstrate an end-to-end sample that consisted of loading data from an SqlServer database by means of the Entity Framework (EF) and hooking the loaded data on a Windows Presentation Foundation (WPF) front end. The requirement for the database were really simple. I just needed a DB that would run on Microsoft SqlServer, contain a couple of tables with relations and some data in them.

I remembered the good old Adventureworks database that is a perfect fit for any database related product demo. There are various possibilities on how to get the Database on http://www.codeplex.com.

I decided to go for the out-of-the-box-ready solution that provides an *.mdf and an *.ldf file that can be attached to an Sql Server, using the following link: http://adventureworksdw2008.codeplex.com/

However, as usual, it was not as easy as I hoped it would be. :-(

When I tried to start the SqlServer on my box, using the SqlServer Configuration Manager, I got an error message stating that “The remote procedure call failed. [0x800706be].

image

It looked like my Sql Server installation was somehow broken. I fiddled around for about 20 minutes and decided that I would just reinstall the SqlServer and not waist more time. Unfortunately, the SqlServer is a piece of software that, in my humble opinion, is not installed in 5 minutes time. As soon as the installer completed, it triumphantly told me that everything installed perfectly EXCEPT FOR THE DATABASE ENGINE!!! *grrrrrr*. That surely pissed me off. Annoyed and disappointed I went to sleep. (Needless to say, it was already very late at night.)

That left me in the awkward situation that I would have to show up at the customer site in the morning and fix my database problem sometime between the getting out of bed and the start of the course that I was about to teach. *sigh*

But then, IT HIT ME!

Since I am a huge fan of Microsoft Azure and deeply appreciate the technology behind it and how it facilitates a developers life, I turned my back on the local SqlServer installation on my notebook and looked up into the skies or, I must say, the cloud. And, alas, the Azure cloud really saved my day!!! Literally 5 minutes later, I had my db running and could even take a coffee break before the course started. *Tataaa*.

In the next section, I will demonstrate how amazingly easy it is to get the Adventureworks db running on Microsoft Azure. Keep in mind that the process that I am going to describe replaces all of these tasks:

  • Providing a virtual or physical machine
  • Downloading an SqlServer Installer
  • Organizing a License
  • Installing the SqlServer
  • Configuring the SqlServer
  • Downloading and Installing Adventureworks

 

Running Adventureworks on Microsoft Azure SQL

The following tutorial is divided into four parts:

1. Prerequisites
2. Provisioning an Azure Sql Database Server
3. Getting and Deploying the Scripts
4. Verifying the Installation

Let’s get started:

Step 1: The Prerequisites

You need 3 things in order to reproduce this example:

1. A Microsoft Azure Subscription
If you do not have one, you can get a free trial subscription for one month and 200 CHF at:
http://azure.microsoft.com/en-us/pricing/free-trial/
2. A computer with an installation of the .NET framework in version 4.
If you run Windows 8, you are set. If you run an earlier version and .NET 4.0 is not installed, you can get it at the Microsoft Download Center: http://msdn.microsoft.com/en-us/vstudio/aa496123
3. The Azure db must be accessible from your client computer. This is a configuration setting in the Azure Management Dashboard. Details can be found below.

Note that currently there are two Azure Portals. The old one under https://manage.windowsazure.com and the new one under https://portal.azure.com. For simplicity, I am going to use the old one.

Step 2: Provisioning an Azure Sql Database Server (Old Portal/manage.windowsazure.com)

After you have created your Microsoft Azure account, go to the management portal: http://manage.windowsazure.com and log in.

On the portal, select SQL DATABASES from the navigation on the left hand side:

image

Once you are on the SQL DATABASES page, click SERVERS on the top navigation:

image

Finally, click the ADD button at the bottom:

image

The CREATE SERVER wizard will pop up, fill in the fields and confirm. Select a region that is close to your location.

image

After some time, it took about 30 seconds for me, your database server is ready to go! Click the little arrow on the right of the server name to proceed to the server settings:

image

On the following page, click CONFIGURE and then click the arrow on the right, where it says “ADD TO THE ALLOWED IP ADDRESSES”. This configuration is vital since it allows your local machine, where the browser is running, to access the database server. IMPORTANT: Do not forget to hit the save button, once completed!

image

Note the server name and the credentials that you specified and we can go on with the next step. Alternatively, you can navigate to the DASHBOARD page and copy the MANAGE URL from there.

Step 3: Getting and deploying the scripts

Navigate to http://msftdbprodsamples.codeplex.com/releases/view/37304 and download the “AdventureWorks2012ForWindowsAzureSqlDatabase” file. Save and extract the zip file.

Open a CMD prompt with administrative privileges and navigate to the directory where you unzipped the package and there into the \AdventureWorks directory.

Enter the following command:

CreateAdventureWorksForSqlAzure.cmd <servername> <username> <password>

Important Hints!
Make sure you enter the name as “username@servername”. Providing the username only can lead to problems during the installation. Additionally, you have to use the full server name, as in: zvnhp88skk.database.windows.net.
Example:

CreateAdventureWorksForSqlAzure.cmd zvnhp88skk.database.windows.net mme@zvnhp88skk.database.windows.net mypassword

image
After roughly 5 minutes, the script should show “Installation Completed”. That’s it, your database is up and running.

Step 4: Verifying the installation

You can use any SqlServer tool, such as Visual Studio or the SqlServer Management Studio to connect to your DB and verify that the data is there. I would like to show a very simple approach using SqlCmd.exe. SqlCmd is shipped with Visual Studio and the easiest way to use sqlcmd.exe is by using the Visual Studio Command Prompt. Run the following command inside the VS command prompt:

sqlcmd.exe –S <servername> –d <db name> –U <username> –q <query>

or, in our case:

sqlcmd.exe –S zvnhp88skk.database.windows.net –d AdventureWorks2012 –U mme@zvnhp88skk –q “select top 10 firstname, lastname from person.person”

If you see 10 records that contain names, the data is there and we are done!

Important hint: Always keep in mind that Microsoft Azure is based on a pay-as-you go policy and you will be charged on a per-minute base for your database instance! If you don’t need it, shut it down!

Getting started with Orchard CMS – The Modules and Features you need

After having installed Orchard as shown in http://www.12qw.ch/2014/03/getting-started-with-orchard-cms-installing-orchard it is very important to know what the first modules are that need to be installed. In this post I will give you an overview about the most important modules/features and why you need them.

The following features help you with questions such as:

  • How can I get Edit buttons on my Orchard pages?
  • How can I influence my Orchard Layout?
  • How can I get table editing functionality in my Orchard HTML Editor.

Widget Control Wrapper

The widget control wrapper is installed by default. Once activated, it will render “Edit” Buttons for all the content sections on the front page. Therefore this:

image

becomes this:

image

Follow these steps to activate the Widget Control Wrapper:

  • Log on to your web site
  • Click the dashboard link at the bottom
  • Click Modules in the navigation
  • Under Features search for Widget
  • Click enable on the Widget Control Wrapper Box

Shape Tracing

The shape tracing feature creates a menu on the web site that allows you to look at the layout structure of the page. Once activated, it displays a small icon at the bottom of the page:

image

The icon opens a menu that lets you analyze all the content element and look at their configuration in terms of Shape, Model, Placement, Template and HTML.

image

TinyMCEDeluxe

TinyMCEDeluxe extends the built-in WYSIWYG HTML Editor with more functionality, such as table editing features. So this:

image

Becomes this:

image 

TinyMCEDeuluxe is a module that needs to be installed from the gallery. In order to do this, navigate to Modules –> Gallery. Search for TinyMCEDeluxe and install the module.

IMPORTANT: Before you activate TinyMCEDeluxe, you MUST deactivate the built-in editor “TinyMCE”.

Getting started with Orchard CMS – Installing Orchard

Orchard CMS is a free Content Management System (CMS) based on Microsoft ASP.NET MVC. http://www.orchardproject.net/

Orchard is very powerful due to its rich extensions in the form of Modules (= Feature Packages) and Themes. It is highly customizable due to its full .NET integration and extensibility.

Orchard can be installed either by downloading with the Web Platform installer or by using WebMatrix. I am going to show you a brief description on how to create an orchard website using WebMatrix.

1. Installing WebMatrix

Microsoft WebMatrix is a free Microsoft web development tool. It focuses on easy creation, configuration, publication and maintenance of websites. The power of WebMatrix lies in the simplicity of its usage and, to my opinion, in the rich template gallery that can be used as a starting point. The gallery contains all the major open source implementations for CMS, Blogging, eCommerce, Galleries and Forum websites.

Examples include: Joomla, Umbraco, MojoPortal, nopCommerce Drupal, Moodle, WordPress, DasBlog and many more.

Go to http://www.microsoft.com/web/webmatrix/ and download WebMatrix.

2. Start WebMatrix

image

3. Create your Orchard WebSite

    • Click “New” –> “App Gallery”
    • Select Orchard CMS and click “Next”

image

  • Click next again
  • Accept the EULA
  • Wait until the download completes.

When the download completes, Orchard will start and open an initial configuration page in the browser:

image

  • Fill in the Site Name, an admin username and password
  • Select your data storage: SQL Server Compact, SQL Server or MySql
  • Select an Orchard Recipe. If you want to create a blog, select blog, if not, select default.
  • Click Finish Setup

And you are done, your website is running on your local machine:

image

From the home website you can access the dashboard via the link at the bottom. The dashboard contains all the configuration options for settings, content, etc.

image

 

In the next post we will explore the features and modules that you need after you have completed the orchard installation:

http://www.12qw.ch/2014/03/getting-started-with-orchard-cms-the-modules-and-features-you-need

2 Comments

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 7: The Sample Application 3

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

Writing the StreamInsight Queries

The sample code to the application can be downloaded here:

Scenario 1: Tweets per Second

As we can see from our pass-through implementation from the last post, we receive a large amount of tweets. In the first query we would like to find out how many tweets we receive per second. This turns out to be pretty easy!

image

We define a tumbling window with a length of one second and just call the count property. StreamInsight uses three different types of windows:

  • TumblingWindow: A fixed length window. The next window begins when the current one ends.
  • HoppingWindow: A fixed length overlapping window. We can define the time interval in which we want the next window. Example: We have a 3 second window that moves in one second hops. So we get updated data every second for the last 3 seconds.
  • SlidingWindow: The sliding window reacts to the input stream and always moves when there is a change in the stream. The window size is defined by two adjacent events.

In order for the example to work, we have to do two more things. First, we need to adjust the consoleObserver. Since the query produces scalar count values (and not TweetItems anymore) we need to adjust the generic type parameter in the DefineObserver() method from TweetItem to long.

image

Next, we need to replace the twitterstream instance with the query for the Bind() call. We now have a query that operates on the data source and bind the observer to the query.

image

That’s it! The output should look somewhat like this:

image

Amazing stuff already, but wait, there’s more!

Tweets per Second grouped by Language

For the next query we are going to group the tweets by language. We first need a data object for the computed output. As a little twist we add a method call that resolves the culture info that we find in the Tweet.

image

Then we continue and write our query. Remember that we have to adjust the generic type parameter of the observer to use the new LanguageSummary type:

image

The query leads to output like this:

image

Nice.

Language with most Tweets per Second

Next, we improve the last query by adding another query that uses the output of the last example as input. We want to know the top 5 languages in terms of tweets per second. We use the SnapShot window here to react in changes in the stream, that is changes in the output of the last query.

image

image

The most popular Tweet every Second

Next, we want to find the most popular Tweet every three seconds. First, we create a new data class:

image

Then we write a query that groups all the tweets in every 3 second window by the number of followers that the user has:

image

image

Note: The numbers after the user name show (Followers/Friends).

In the next step we improve the last example by adding the Friendcount. We want to know the person in every three second window that has most followers AND most friends. We implement this by writing two different queries for followers and friends and then join them together using a StreamInsight join operation.

Let’s start by writing the most friend query first. It is similar to the followers query:

image

Now we use the join to join the queries on the user name:

image

And that is it. If we run the sample we find out who the most popular person on twitter is ever three seconds:

image

An interesting observation that we can make looking at the data is that in some windows, e.g. 06:21:57, there is no event. This means that at that point the person with most tweets and most friends was not the same person.

The join is a very powerful operation that can give us deep insight into the data.

If you want to play around and try out more complex queries, look at the resources in the first blog post of the series, especially at the LINQPad Samples and the Hitchhiker guide.

I hope you enjoyed this series on Microsoft StreamInsight. If it provided substantial value to you, feel free to donate. :-)

3 Comments

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 6: The Sample Application 2

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

The sample code to the application can be downloaded here:

Putting it all together

Now that we have written our TweetItem, TwitterStream an Unsubscriber class, we can put it all together and write the actual StreamInsight code. We start by implementing the main method.

First we define the StreamInsight Server. In the Create() method we specify the name of the StreamInsight instance as we defined it during the installation. Then we create a StreamInsight application that will hold our data sources, sinks and queries.

image

Next, we define our data sink. We write a method that prints the event to the console. There is an IF statement that only prints the stream events that we are interested in and omits the CTI’s. We implement it as a generic method with a type variable for the payload. This allow us to reuse it when we have different queries with different output POCOs:

image

Following the StreamInsight 2.1 approach we can create a sink around the method above and hook it to the application in one line. Note: You have to include the Microsoft.ComplexEventProcessing.Linq namespace in order to see the DefineObserver() extension method.

image

However, it is one line but to my opinion it is hard to understand. Let’s pick it apart and analyze what is going on:

image

We use the static Create() method from the observer class to create an Observer instance that can observer PointEvent<object> and we point it to the ConsoleWritePointNoCTI method that accepts an element of type PointEvent<object> as a parameter.

Next, we create our data source from the TwitterStream.cs class.

image

Here as well, we can write it in one line but it is even more complicated than the last statement. Lets look at what is going on:

image

We can see in the code that we begin by creating an instance of our TwitterStream class. Remember that this is the class where we implemented the IObservable<TweetItem> interface.

Then we use the DefineObservable() extension method from the StreamInsight assembly to convert the IObservable into an IQbservable. From there, we convert the IQbservable into an IQStreamable. In this call we define the structure of our PointEvent items and instruct StreamInsight to add CTIs to the stream.

Finally, we create a binding that connects the data source and the data sink and we call the Run() method on that binding. In this first example, we do not yet use a query and just run all the data that we receive in the source directly to the sink.

image

And that’s it! Hit F5 and HERE WE GO!

The output should look something like this:

image

Note: The question marks come from languages with characters that can not be printed by the standard console.

Bravo! You can now receive live Twitter data processed by StreamInsight!

Next week, we will write our first queries and process the data between the source and the sink.

1 Comment

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 5: The Sample Application 1

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

The sample code to the application can be downloaded here:

Writing the Sample Application

In this post we are going to write a sample application from scratch. We start by installing StreamInsight and then jump into Visual Studio and start coding the project. First, we are going to use a library to get access to the Twitter feed. As soon as we have the basics we elaborate on some interesting queries on the Twitter data stream.

Installing StreamInsight

First, we need to download the StreamInsight installer executable. We can find it in the Microsoft Download Center under: http://www.microsoft.com/en-us/download/details.aspx?id=30149

We need the StreamInsight.msi Installers corresponding to your target platform bitness.

During the installation you will be asked if you would like to create a StreamInsight instance. Do so and give it any name you like. In this example I am going to name it “Thomas”.

Creating the Visual Studio Solution

Now, let’s fire up Visual Studio. I am using VS2012 but you can use VS2010 as well. We make it simple and create a new console project. Then we need to add the Microsoft StreamInsight assemblies. Add assembly references to the following files:

  • Microsoft.ComplexEventProcessing: Found in the StreamInsight install directory: C:\Program Files\Microsoft StreamInsight 2.1\Bin\
  • System.Reactive: Found under .NET 4.0
  • System.Reactive.Providers: Found under .NET 4.0

Your References should look like this:

image

Next, we integrate Twitter into our solution. We use a handy library named Tweetinvi and install it via NuGet Package Manager. Maybe you need to install the NuGet Package Manager in your Visual Studio, if it is not available. Right-click on the References folder in the Solution Explorer and select “Manage NuGet Packages”. Do an online search for “Tweetinvi” and hit “install”.

image

First we are going to write a data class that will be our payload object. We call it TweetItem and fill it with a couple of properties. For the sake of simplicity, we restrict our item to the following fields: User, Text, CreationDate, Language, Followers, Friends.

image

In our TweetItem class we provide a constructor that accepts an object that implements the ITweet interface and converts it into a TweetItem. The ITweet interface can be found in the Tweetinvi Library and contains all the fields that we get for each tweet.

 

Creating the Twitter Observable

Next, we create the class that we use as our data source and feed into StreamInsight. Create a new class called TwitterStream.cs and implement the IObservable<TweetItem> interface. The interface contains a Subscribe() method:

image

First, we are going to add a list of IObserver<TweetItem> objects. Then we add code to the Subscribe() method that adds the observer parameter to the observer list.

image

The result of the subscribe method is an object that implements IDisposable. The subscriber can use it to cancel his subscription. The subscription will end when the subscriber disposes the object that he received upon subscription. We implement a small Unsubscriber class that implements IDisposable and keeps references to the observer list and its own observer instance. Upon disposal, it removes its own reference from the observer collection.

image

Next we need a Start() method that connects to twitter and starts receiving Tweets. In order to receive Tweets from Twitter you will need to register on the Twitter website and get the following four credential keys: UserKey, UserSecret, ConsumerKey, ConsumerSecret. You can start on: https://dev.twitter.com/docs. Once you can login to dev.twitter.com you can create an application and will get your authentication information.

There is a free subscription to Twitter that will give you one percent of all the worldwide Tweets. Once you have your credentials, we can implement the start method:

image

The start method uses the Token and SimpleStream classes from the Tweetinvi namespace. Note that the SimpleStream class has nothing to do with StreamInsight. In the StartStream() method we provide a lambda expression that points to the OnNewTweet(tweet) method that we will define next. The OnNewTweet() method is used to map the Tweets into TweetItems and forward each Tweet that we receive to all the items in our observer collection:

image

Finally, we implement the constructor. We initialize the myObservers collection and we call the Start() method. We execute the Start() method on a background thread using the Task Parallel Library (TPL).

image

In the next post we put it all together and write our first query. Stay tuned!

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 4: Reactive Programming

Reactive Programming with the Rx Extensions

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

In this chapter we are going to look at the StreamInsight 2.1 approach. We analyze and try to understand the interfaces that we met at the end of the last post.

image

We start by looking at where the different interfaces have their origin. This will give us clues on the functionality that they provide:

image

IEnumerable<T>: The IEnumerable interface comes from .NET 2.0 and has been there forever. We use it to describe an enumerable collection of items. All the common collections in .NET, such as Lists, Arrays, Sets, etc. implement the IEnumerable interface. It enriches the collection with functionality and is required if you want to “foreach” over a collection.

IQueryable<T>: IQueryable is the dual to IEnumerable. It serves the same purpose, describing a collection that contains enumerable items. The difference lies in the implementation. IQueryable can be used to access a remote collection of items. The interface contains an abstraction of the functionality of IEnumerable that can be projected onto any remote system, if there is a matching query provider available.

IObservable<T>: The IObservable interface can be found in the .NET 4.0 namespace. It fits together with the IObserver<T> interface and can be used to push data from the observable object to the observer object. We use it to implement publish-subscribe.

IQbservable<T>: Probably the interface with the strangest name ever, right? By the way, the Q stands for “Query”, as in IQueryable. IQbservable is to IObservable what IQueryable is to IEnumerable. This means that we can use IQbservable to implement the push paradigm with an observer that resides on a remote system.

IQStreamable<T>: IQStreamable is the interface that StreamInsight uses to represent a data stream.

Let’s look at the interfaces from a functional viewpoint:

image

On this image we can see how the interfaces are related in terms of PULL vs. PUSH and LOCAL vs. REMOTE.

Lets take a closer look at the difference in implementation of the PULL and the PUSH versions of the interfaces.

image

We can see that IEnumerable defines a GetEnumerator() method. The enumerator contains methods to navigate through the items in the collection.

On the other hand, the IQueryable interface is much more abstract. It provides an expression and a queryprovider. The expression is a statement represented in the form of an expression tree. An expression  tree is an abstract construct that can represent a query such as the where clause in a LINQ statement.

The idea is that due to the abstract implementation of the expression, it can be sent to a remote system. On the remote system. The query provider is responsible for translating the expression into the language of the remote system. If we think about SqlServer, this might be SQL syntax.

We can see that the relation between IObservable and IQbservable has exactly the same purpose:

image

The IObservable interface provides an OnNext() method. Recall that we use it for publish-subscribe purposes. The OnNext() method is the place where the Observable pushes the data to the Observer. The IQbservable represents the logic in an expression and a query provider for remote execution.

WPF Samples: Real World WPF Line-of-Business (LOB) Applications

Have you ever wondered what WPF LOB Applications are out there in the wild?

Pete Brown published an interesting collection of real world WPF applications called the Book of WPF. It is already a couple of years old but still interesting to look at:

http://10rem.net/blog/2010/02/10/the-book-of-wpf

You can download the PDF directly here:

http://archive.msdn.microsoft.com/Project/Download/FileDownload.aspx?ProjectName=WPFPresents&DownloadId=8223

Somehow I got my hands on the following Power Point Deck. I could not find it anymore. It contains some of the projects from the Book of WPF in a more shiny way:

WPF

Very interesting stuff:

image

image

image

image

image

Using Complex Event Processing (CEP) with Microsoft StreamInsight to Analyze Twitter Tweets 3: StreamInsight Programming

Note: This post is one of a series, the overview can be found here: Complex Event Processing with StreamInsight

In this part, we ar going to look at the Streaminsight programming model. 

What do we need?

  • Visual Studio. In order to use StreamInsight there is always some .NET coding required.
  • A hosted StreamInsight server. The server can be installed from the StreamInsight executable as a windows service. (about 10MB)
  • The StreamInsight Installer executable. It contains the server component and the assemblies that need to be referenced in Visual Studio.
  • An SqlServer license. For productive use of StreamInsight, we need a valid SqlServer license.

Important to note:

  • We do not need an SqlServer instance in order to use StreamInsight
  • StreamInsight is more a framework than a complete product. There is always some coding involved.

StreamInsight Artifacts

This section will explain the artifacts and concepts that need to be understood in order to start developing the StreamInsight solution.

image

A StreamInsight solution always consists of the items listed below. In the demo application we will see all of these again in our code.

  • A StreamInsight server. The server usually runs as a windows service and can provide multiple instances where we deploy our data sources, sinks and queries.
  • A StreamInsight Application. The StreamInsight application is a logical container in the server that contains our artifacts.
  • A data source. The source can either be standing, e.g. historical data or it can be moving data from a live data stream. implementation is the same.
  • A data sink. The place where StreamInsight puts its output.
  • A query. The query is written in LINQ
  • A binding. The binding connects the query to a sink.

StreamInsight Event Types

The data items that we feed from our .NET code into and out of the StreamInsight engine are always generic types with a POCO payload object. These generic types contain temporal information such as start and end times plus the POCO payload object. We differentiate three different types according to the nature of the event:

  • PointEvent<T>: A point event is an event with a single timestamp. It can be used for events that occur exactly once and would be visualized in the form of a point.
  • IntervalEvent<T>: An interval event is an event with a start time, an end time and a fixed duration larger than 0.
  • EdgeEvent<T>: Edge events are as well defined by a start and an end point. The difference to IntervalEvent is that as the start time passes and the interval begins, we do not yet know either the duration nor the end time of the event.

Another event type in contrast to the data events are Current Time Increments (CTIs). StreamInsight needs to have CTI events in the stream that it is going to analyze. The developer has to define how CTI are inserted into the stream. CTIs mark constant time windows inside the stream:

image

StreamInsight needs CTIs to process the stream for the following reasons:

  • Stream processing: A CTI “closes” the current timeframe. Let’s make an example: If we work with 1-minute CTIs, when the 12:05 CTI arrives, StreamInsight will process the 12:04 – 12:05 window. If we have late arriving events that belong to the window but arrive after the 12:05 CTI, StreamInsight will ignore them.
  • Stream sync: If we evaluate data from multiple streams, StreamInsight will synchronize the two streams based on their respective CTIs.
  • Non-events: We can look for missing events in CTI windows. We could use this approach to implement a monitoring solution that regularly checks on a heartbeat signal.

 

Getting the data in and out of StreamInsight

In order to connect StreamInsight to data sources or data consumers we need to start programming.

StreamInsight knows 2 different implementation approaches. The first one is the adapter approach. It has been there forever and can be used for complex input and output. As of StreamInsight 2.1, the adapter approach is often called the legacy approach.

The second approach is a feature of StreamInsight 2.1 and relies on existing classes from .NET or from the .NET Reactive Extensions (Rx Extensions). The idea is that if we already have a class that implements the interfaces IEnumerable<T> or IObservable<T> we can hook these into our processing.

 

The Adapter (Legacy) Approach

The adapter approach is pretty straightforward. We use the IInputAdapter<T> and IOutputAdapter<T> interfaces and implement them in our classes. Then we provide a config and a factory class and instruct StreamInsight to request an adapter from the factory.

image

 

The StreamInsight 2.1 Interface Approach

The interface approach is said to be much easier. I personally think that it is way harder to understand for a beginner. There are a lot of interfaces and once you grasp the purpose of each one, it starts to get more understandable.

image

We have three groups of interfaces that we use depending on the nature of our data source:

  • IEnumerable<T>/IQueryable<T>: If we have a collection as a data source.
  • IObservable<T>, IQbServable<T>: If we have a stream as a data source.
  • IQStreamable<T>: IQStreamable is the interface that StreamInsight understands and describes a data stream.

StreamInsight provides extension methods that let us create an IQStreamable object from any of the other interfaces. This is very convenient for us. In short, this means that we have to create or convert our data source into IEnumerable (for PULL) of IObservable (for PUSH) and can then convert this object in to an IQStreamable.

I will go into the details of all the interfaces in the next chapter.