The Pitfalls of Mapping the Entity Framework to Stored Procedures

A Guide

Too much of the literature about the Entity Framework contains directions for mapping directly to Stored Procedures. The way it is presented, it seems easy enough. However, attempting to implement this strategy beyond a simple example quickly becomes unnecessarily difficult.

Let’s say you would like to implement a Model First pattern whereby all the entities’ create, update, and delete (CUD) methods were mapped to Stored Procedures. For cases where the database already exists or where using Stored Procedures is a predetermined constraint of the project, this pattern enables your .NET code to mimic typical LINQ to Entities code. Ideally, the mappings would be mostly abstracted away into the Entity Framework definition file. Once mapped, developers would not have to pay much attention to this implementation detail.

Unfortunately, there’s a vast difference between what you can do with the Entity Framework in a simplified demonstration and the restrictions meeting you in practice. Below is a guide that covers the pros, cons, and restrictions of the “EF-to-STP” approach.

Advantages

There are some advantages to using Stored Procedures. Mapping them in Entity Framework potentially produces the best features of both technologies.

Legacy Databases

When a database already exists that has implemented a great deal of logic in Stored Procedures, it can be reused.

Abstraction layer

Stored Procedures form a layer “underneath” the .NET code. Behind this abstraction layer, .NET developers do not need to understand implementation details.

Granular Security

Every Stored Procedure can have user-specific permissions.

Developer Skillset

Business logic implementation can be performed by a database developer whose skillset is strong in Stored Procedures.

“Advertised” Restrictions

Pluralsight.com has several video resources with tips about implementing the Entity Framework with Stored Procedures, including these three which I found helpful:

Below are some of the limitations mentioned in the videos:

All CUD Stored Procedures must be mapped for an Entity

For each entity, if any of the Create, Update, or Delete functions is mapped, then all 3 should be. For example, if you only map the Delete function to a Stored Procedure but at runtime your .NET code causes an Update to occur on that entity, it will throw an exception. It will not use the default Entity Framework functionality for the Update call.

Parameters can be mapped only to entity properties

You cannot map the parameters to a scalar value or a function. For example, you cannot map a Date parameter to DateTime.Now.

Stored Procedures that return entities have change tracking by default

In contrast, Stored Procedures that return complex objects do not support change tracking.

Insert Procedures must return SCOPE_IDENTITY

In the last line of your Insert Stored Procedures, include the line “Select SCOPE_IDENTITY() as ID” so that Entity Framework can push the newly generated record ID back into memory.

No support for multiple result sets

Limitations Found Through Experience

The biggest issue with using EF-to-STPs is that the developer of the Stored Procedures probably did not realize the database would be consumed this way. To do it right, a great deal of consistency is needed. There must be 1 Stored Procedure for Insert, 1 for Updates, and 1 for Deletes per entity. The Stored Procedure developer may have optimized for different things, such as readability or a reduction in round-trips to the database, instead of for a standard pattern that can be used by a high-level Framework. The below tips can be helpful as a guide to developing Stored Procedures if they are not yet finished.

Do Not rely on default parameters

You must specify exactly 1 parameter per entity column for select queries. Similarly, for an update query, every column must have a parameter in the Stored Procedure, and you must map every parameter in the Entity Data Model. Therefore, Default parameters have almost zero purpose for CUD Stored Procedures in our scenario.

Do Not rely on output parameters

Output parameters in Stored Procedures used for CUD cannot be mapped in the Entity Data Model.

Do Not rely on custom or table parameters

The Entity Framework can consume Stored Procedures that have table parameters. However, they require more of an ad hoc coding structure. They do not fit well into our EF-to-STPs pattern.

No ability to include referenced tables

The Entity Framework can utilize Function Imports to return entities from Select Stored Procedures. However, the ability to include reference tables is not built-in.

Extra Stored Procedure logic can fool Entity Framework

Scenario: An Update Stored Procedure performs additional logic to update another table before updating the mapped entity’s table.

Outcome: The Entity Framework got confused by how many records were updated and rolled-back the update.

Ensure returned data types are consistent

Scenario: A Select Stored Procedure uses IF statements and CASE statements to determine what Select query to use. Reading through the code, it’s difficult to detect, but one path returns Integers while another returns Bits.

Outcome: The Entity Framework generates metadata using one path and throws a cast exception at runtime.

Proceed with Caution

The Entity Framework is a great tool, especially when it is used with LINQ queries to the database. While Microsoft and its partners describe how to map the Entity Framework to Stored Procedures, it only works in simple cases or where the Stored Procedure developer follows very strict standards. In most cases, it is not practical.

Still, the Entity Framework can very well be used to query Stored Procedures with a more verbose approach. It requires more code than EF-to-STP mapping described above. However, it typically requires less code than reverting to older technologies like ADO.NET. Just stay away from the mapping.

Disclaimer: Most of my experience in this area is with Entity Framework Version 4, but I believe it applies to Version 5 as well.

Advertisements

How to Generate .NET Classes from a Non-.NET Data Feed

Most people do not realize how much data is available on the web via APIs. Indeed, we .NET programmers tend to be a breed that ignores the trendy new data feeds that are available. Perhaps it’s because it is intimidating to try to interact with sites written in PHP or Ruby on Rails or maybe it’s because the only examples anyone ever shows are for Netflix or Twitter APIs (2 APIs that are not particularly useful for an Enterprise Developer). Now is the time to expand your horizons. As more and more data becomes available, the usefulness increases for all types of applications. I aim to broaden your awareness of the entire domain of public web services (APIs) and show you that

  • .NET can be a great client coding-language for any standard web API
  • There are great online resources to find available APIs.

 

Table of Contents

Before I dive head first into all the details, here is an outline of what I will cover and the basic steps involved:

  1. Purpose/Intention
  2. Choosing an API
  3. Review Documentation
  4. Determine a Sample Query
  5. Generate Classes at Json2CSharp.com
  6. Paste Generated Classes into New Application
  7. Massage Generated Classes
  8. Reference Json.NET
  9. Retrieve Data with a WebClient Object
  10. Display Results within Application

 

Purpose/Intention

At this point, you may be asking yourself, “why do I care about Data Feeds, APIs, and public Web Services”? You should care because it is the technology through which online companies share their data. If you think it might be worthwhile to someday automatically retrieve the weather forecast, stock prices, sports scores, site analytics, etc. and make logical decisions based on the data, then pay attention because the steps that follow are how you get started. A well-known example of a website using a public APIs is Expedia.com, which retrieves commercial flight and hotel information from multiple providers based on a user’s travel criteria. There’s very little stopping us .NET developers from gathering together multiple APIs in a similar fashion.

 

Choosing an API

The first step of connecting to an API is to choose which one you will connect to. If you already know, you need to find out more information about it. To do this, I used ProgrammableWeb.com, an online directory of public-facing APIs. When I started the exercise for this blog post, I did not know which API I wanted to test with, so I just clicked on API Directory | Newest APIs. As tempting as it was, I chose not to use the Stack Overflow API, because it is already built in .NET and is therefore disqualified from this blog post. Instead, the API for the Khan Academy caught my eye.

In case you haven’t heard of it, Khan Academy is a non-profit organization that provides a wide range of training videos and courseware for free online.

The Khan Academy API is perfect:

  • It’s not a .NET service written in WCF (it’s Python)
  • I hear about the site all the time. It’s sexy right now.
  • It follows a web standard, REST formatted in Json.

By clicking the Khan Academy link in the Programmable Web directory, I was eventually taken to the Khan Academy API documentation site.

 

Review Documentation

Many large websites have thorough documentation about their APIs. Still, there is a wide range of information that you may come across when researching them. Some have client-side examples in .NET and others even have 3rd party libraries (e.g. MailChimp) specifically written for them. Khan Academy has a nifty tool called their API Explorer, which allows you to click on different types of REST queries and see example responses. . I’ve seen similar tools on other sites too, such as Yahoo.

 

Determine a Sample Query

To start creating our .NET client application, we need to determine a sample query and retrieve response data. I’d like to generate local, .NET classes to consume the information sent back from Khan Academy.

There are a couple different ways of thinking about this:

If I know specifically what type of information I will be using, I can look for documentation on how to retrieve that narrow result set. In this case though, I want to start with as many classes as possible, to fill out my .NET solution with a large portion of classes.

The playlists/library/ query is great because it returns nested results. So, for example, it has information for playlists, with sub-information about videos, tags, etc.

Having a sample response like this is half the battle, and it’s not that difficult to get for REST services.

 

Generate Classes at Json2CSharp.com

Once we know what sample query we are going to use, we continue to our 2nd big step, where we either paste a URL or Json results into a website named json2csharp.com.

This website converts the sample response data that we entered into .NET class definitions. With this step, we are letting the Json2CSharp website perform a significant step of the process for us automatically.

Why do we go through the effort to generate .NET classes like this?

  • It enabled compile-time checking and intellisense
  • Design-time use in Web Forms Gridviews, etc.
  • Can be used in older versions of .NET (earlier than .NET 4), which do not yet have dynamic objects.

 

Paste Generated Classes into New Application

Now that I have generated .NET classes, I will copy them into my Windows Clipboard (Ctrl-C) for later use.

Let’s keep things simple by creating a brand new Web Application. In Visual Studio 2012, select File | New | Project. Then select an ASP.NET Web Forms Application.

 

With the new application in place, let’s add the .NET classes into the solution.

First, add a class file to the project.

 

In this file, paste (the .NET classes that are in your Clipboard) over the default class. As a quick sanity check, you should be able to successfully compile the solution.

 

Massage Generated Classes

Json2CSharp sometimes struggles with ambiguous responses. As a result, it generates duplicate class definitions as is true in our case.

Still, it’s nice that the class generator got us part of the way toward our final code. Let’s massage our classes to remove any classes that have numbers on the end. Also, switch any reference to the duplicates back to the primary class.

Delete: Item2, DownloadUrls2, Video2, Playlist2, DownloadUrls3, Video3, Playlist3

Alter: References to Item2 -> Item, References to Playlist2 -> Playlist, Reference to Playlist3 -> Playlist

 

Reference Json.NET

In order to deserialize Json results into our generated classes, we need to use another 3rd party tool named Json.NET. To add a reference to this library, we can perform either of 2 methods:

Download and Add Reference

  1. Browse to Json.CodePlex.com
  2. Download the latest version as a .zip file
  3. Extract the relevant version of the Newtonsoft.Json.dll
  4. Add a reference to the dll

Install with NuGet

  1. In Visual Studio, go to Tools | Library Package Manager | Manage NuGet Packages for Solution
  2. Click Online in the left panel
  3. Click Json.NET to highlight that package
  4. Click Install

 

Retrieve Data with a WebClient Object

At this point, we’ve got the framework setup in our solution to store strongly-typed representations of the Khan Academy data. Next, we need to write the code to retrieve that data.

Here is the snippet I put in the Default.aspx.cs file to automatically retrieve the data and format it with Linq.

public static List<Playlist> GetKhanVideos()
		{
			var client = new WebClient();
			var response = client.DownloadString(new Uri("http://www.khanacademy.org/api/v1/playlists/library"));
			var j = JsonConvert.DeserializeObject<List<Item>>(response);

			List<Playlist> playlists = new List<Playlist>();
			playlists.AddRange(j.Select(i => i.playlist));
			playlists.AddRange(j.Where(k => null != k.items).SelectMany(i => i.items).Select(i2 => i2.playlist));
			playlists.AddRange(j.Where(k => null != k.items).SelectMany(i => i.items).Where(k2 => null != k2.items).SelectMany(i2 => i2.items).Select(i3 => i3.playlist));

			return playlists.Where(p => null != p).ToList();
		}

 

Display Results within Application

In our last step, we want to see the output of our query, so let’s leverage the drag-and-drop ability of Web Forms to easily visualize the data.

  1. Open Default.aspx in Design View
  2. Using the Toolbox, add an ObjectDataSource
  3. Configure the DataSource
  4. Choose WebApplication1._Default as the business object
  5. Choose GetKhanVideos() as the Select Method
  6. Using the Toolbox, add a GridView
  7. Configure it to choose the above Data Source
  8. Many of the fields will be empty or gibberish, so let’s remove several columns:
    1. backup_timestamp
    2. hide
    3. init_custom_stack
    4. ka_url
    5. kind
    6. standalone_title
    7. topic_page_url
    8. url
    9. youtube_id

To see the new application in action, press F5 to run it.

 

Conclusion

With the help of a few 3rd party tools, retrieving and displaying any REST-based API in .NET can be easy. Not only that, but it’s going to get even easier. In Scott Hanselman’s ASPConf keynote, he showed an extension that is being developed by Mads Kristensen for Visual Studio 2012 that would eliminate several of these steps. It allowed an option in Visual Studio to “Paste JSON as classes,” thereby eliminating the need for the class-generation website. Microsoft realizes that the trend of creating and leveraging public APIs is not going away so they are doing something about it. And so should you.

 

Disclaimer: This product uses the Khan Academy API but is not endorsed or certified by Khan Academy (www.khanacademy.org).

Don’t Disrespect the Web.Config – ELMAH

This blog series has focused on simple changes that can be made to a .NET solution’s web.config in order to enhance the development environment, enhance security, and improve troubleshooting capabilities.

You can find previous posts here:

  1. Use web.config transforms to keep track of development versus production settings
  2. Encrypt important configuration sections for security
  3. ELMAH – Error Logging Modules and Handlers

This is the 3rd and final post of the series in which we discuss ELMAH, short for Error Logging Modules and Handlers. I am definitely not the first to write about this, but it is such a useful tool that fits snuggly into the web.config that I had to include it in the series.

 

What is it and why do I care?

First, let’s explain what ELMAH is. It is an open-sourced component that can be easily added to a .NET project for the purpose of logging and notifying developers of unhandled exceptions. What is an unhandled exception? It’s an error in code that a web application cannot respond to, often resulting in a “Yellow Screen of Death.”

ELMAH does not, by itself, rid your application of a Yellow Screen of Death, a screen that causes much frustration among users of your application. Instead, it automatically logs the details of the exception, and the stack trace at the time of the exception, and it can even email the development team that something bad happened.

Using ELMAH has become the standard for any project that I work on. It’s just so darned useful for troubleshooting issues and doing great customer service.

 

Typical Workflow of a User Encountering an Error

Most of the time, users encountering an error do not immediately send an email to support. If it’s a public website, the user might get immediately discouraged and leave the site. If it’s an Intranet website, one that users must use to perform their jobs, then he or she might back up and try it again a couple of times before giving up:

  1. Attempt to perform job function
  2. Curse at screen when exception occurs
  3. Try again
  4. Complain to boss when the same failure occurs
  5. Contact support
  6. Wait
  7. Lose faith in the developers of the website, the support team, and anyone else who is guilty by association

 

Workflow when Error Notifications are Involved

With ELMAH in place, it is easy to short-circuit the workflow and keep users happy. You can begin to troubleshoot the issue before the user has even contacted the support team.

  1. User attempts to perform job function
  2. User curses at screen when exception occurs
  3. Development team member is notified that exception occurred along with details
  4. As the user attempts to try again, the development team member can reach out to the user

    Hello [username],

    I work on the support team for [name of web application]. Our system automatically notifies us when users run into an error that it does not know how to handle, and we are aware that it affects your ability to continue through the application.

    We do not yet know exactly what the problem is, but are working to find out more information and resolve the issue quickly. I will let you know as soon as this is fixed.

    In the meantime, it would help us to resolve this more quickly if you could tell me [what steps you were performing when this crashed].

    Lastly, I know it is less than ideal, but you might try to [perform your job through this work-around or alternative solution] until I get back in touch with you.

    Thank you,

    [Nathan Stuller]

    [Title]

  5. Often, by the time the user would have contacted support, the issue is already fixed

 

Being proactive makes a serious impression on users (and bosses). I’ve used this technique before to reach out to customers about exceptions that they didn’t even notice. It reduced my stress level by confirming that it was a low-priority issue and also allowed me to engage with a customer about my product.

 

How to Get Started

The first step is go to the ELMAH homepage. There you will find the 2 most important links to enable this setup:

  1. Download links to the ELMAH dll you must include in your project
  2. A sample Web.config file (on the same page)

 

The Real Steps

  1. Download the appropriate binary to your web application folder.

  1. Open the zip file. Browse into the “bin\net-2.0\Release” folder and extract the Elmah.dll into your Libs folder.
  2. In Visual Studio, we need to make a reference to this dll from your main web project.

 

  1. Next, you need to add entries into the web.config to tell your project how to use ELMAH. The website has lots of information on how to do this, but I will try to simplify it for the common example of notifying developers by email. First, you need to add this line inside the <configSections> area of the web.config:

  1. Next, add the following line under the <httpModules> section. This may need to be altered depending on which version of IIS your web server is using:

  1. Lastly, add the elmah section (usually at the end) inside the <configuration> section:

There are a host of configuration options you can set to enable ELMAH to do exactly what you want.

 

Wrapping Up

I hope this 3-part blog series has helped you identify simple improvements that can be made to the web.config file. ELMAH, in particular, helps me delete my clients and since it is so simple to implement, to me it is a no-brainer.

Don’t Disrespect the Web.Config – Encryption

As stated in an earlier post…

There are 3 things every public website should be doing with their web.config

  1. Use web.config transforms to keep track of development versus production settings
  2. Encrypt important configuration sections for security
  3. ELMAH – Error Logging Modules and Handlers

In this post, we’ll discuss how to encrypt sensitive sections of the web.config so passwords and other information cannot be easily read by those who gain access to the file.

Why is it important?

Encrypting sensitive sections of the web.config is important because they are just that, sensitive. Think about your production web.config file. It may contain all sorts of data that you would not want to be accessible. There are often passwords for SQL database connections, passwords to an SMTP server, API Keys, or critical information for whatever system is being automated. In addition, web.config files are usually treated as just another source code file. There are probably many versions in your source control system right now. That means, any developer on the team, or more accurately anyone with access to the source code, can see what information is in the web.config file.

In many cases, storing passwords in a web.config is itself unnecessary and should be avoided. However, we know it is all too easy to fall into the trap of placing them in this flexible, convenient file. Therefore, at the very least, certain sections should be encrypted so they cannot be easily read or used for evil.

Encrypting ConnectionStrings

In our example, we will encrypt two typical configuration sections: ConnectionStrings and AppSettings on a Windows 7 development machine.

Follow the below steps:

1. Open a command prompt with elevated, Administrator, privileges:

2. At the command prompt, enter:

cd “C:\Windows\Microsoft.NET\Framework\v4.0.30319”

3. Now enter the following to encrypt the ConnectionStrings section:

aspnet_regiis.exe -pef “connectionStrings” “C:\WebApplication1\WebApplication1”

In this case, C:\WebApplication1\WebApplication1 is the directory where our web.config is located.

4. Enter the following to encrypt the AppSettings section:

aspnet_regiis.exe -pef “appSettings” “C:\WebApplication1\WebApplication1”

For reference on all the command-line options of aspnet_regiis.exe, refer to this MSDN page.

Decrypting ConnectionStrings

Of course, it is possible you might need to be able to read the original, unencrypted, data at a later time. To access that information is easy. Simply perform the previous steps but use the command-line option “-pdf” to decrypt the important sections.

Deployment

Deploying your web application with encrypted web.config sections is simple, but it may not be obvious. This StackOverflow answer explains the steps best. Generally, any server or development machine that uses the same encrypted web.config data must use the same RSA key pair, which can be exported using the aspnet_regiis tool.

Before and After

There you have it. You have successfully encrypted 2 sections of your web.config file. Take a look below to observe the before and after results:

Before

<connectionStrings>

<add name=ApplicationServices connectionString=data source=.\SQLEXPRESS;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|\aspnetdb.mdf;User Instance=true
providerName=System.Data.SqlClient />

</connectionStrings>

After

<connectionStrings configProtectionProvider=RsaProtectedConfigurationProvider>

<EncryptedData Type=http://www.w3.org/2001/04/xmlenc#Element
xmlns=http://www.w3.org/2001/04/xmlenc#>

<EncryptionMethod Algorithm=http://www.w3.org/2001/04/xmlenc#tripledes-cbc />

<KeyInfo xmlns=http://www.w3.org/2000/09/xmldsig#>

<EncryptedKey xmlns=http://www.w3.org/2001/04/xmlenc#>

<EncryptionMethod Algorithm=http://www.w3.org/2001/04/xmlenc#rsa-1_5 />

<KeyInfo xmlns=http://www.w3.org/2000/09/xmldsig#>

<KeyName>Rsa Key</KeyName>

</KeyInfo>

<CipherData>

<CipherValue>fK275KFHx9RKip16DTpwxLi4AHpvCpat4S3edgsDwco9PgudsMKc1qAyh9qNt2y+90qV4QIzyZXm8j27UV5J+R5rNruMUOROLWzVt8qkRYRM3ADoiCi5BJh2SsjE0guGXFbufZDgRpPFV5bstgZSBPYNiYXQF/aOLyQjPCE8VDo=</CipherValue>

</CipherData>

</EncryptedKey>

</KeyInfo>

<CipherData>

<CipherValue>CSdausUH7yWcY8t1sPUqiCooYreEauzi4t33gVJuWYcfhspsguTchJjwthUTMLqnulYRmCu8ZnhrVBepQo7PHO/4k5mwo3s46TsgFddvvUlyY/EDQf047LG0pocBDxL3MgIGf3b+atoG29Jg0Wnhj+M6urYG55Ko4nGp36JILQptlEn+sqCl2sQ99izykXtRWP7kC4tldO+YvBuZ7x8fyGoANwSKQFo7cH+dbydvCkRvaFQsRATdsQKGmSrXwIlkoNvxFb1CBPx0qDenyCs+vO4QyF2CZ8QB+UIJzA8EL7W/FovH5zDczjXQWTsFSmsI+vSojl9G9jSVLJFbwOpQBLIKxfximl5r</CipherValue>

</CipherData>

</EncryptedData>

</connectionStrings>

In a future blog post, we will discuss the 3rd party component ELMAH, which is vital to being notified when your users encounter exceptions in your web application.

 

Update 07/24/2012
It is possible to combine encryption with web.config transforms. I know this will work as I have done it before.

In my experience, I’ve done the following. I had to add an RSA section at the top of my web.config. For me, this went into my web.config.release as I did not encrypt my default/development web.config:

<configProtectedData defaultProvider=”MyRsaProtectedConfigurationProvider” xdt:Transform=”Insert”>
<providers>
<add name=”MyRsaProtectedConfigurationProvider”
type=”System.Configuration.RsaProtectedConfigurationProvider, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL”
keyContainerName=”NetFrameworkConfigurationKey_viternuscom”
useMachineContainer=”true” />
</providers>
</configProtectedData>

Also, the web.config sections need to be replaced at the section level. You cannot replace the name or connection string attributes. So, you can use something like the below configuration to replace the whole connection strings section in each environment:

<connectionStrings configProtectionProvider=”MyRsaProtectedConfigurationProvider” xdt:Transform=”Replace”>
<EncryptedData Type=”http://www.w3.org/2001/04/xmlenc#Element&#8221;
xmlns=”http://www.w3.org/2001/04/xmlenc#”&gt;
<EncryptionMethod Algorithm=”http://www.w3.org/2001/04/xmlenc#tripledes-cbc&#8221; />
<KeyInfo xmlns=”http://www.w3.org/2000/09/xmldsig#”&gt;
<EncryptedKey xmlns=”http://www.w3.org/2001/04/xmlenc#”&gt;
<EncryptionMethod Algorithm=”http://www.w3.org/2001/04/xmlenc#rsa-1_5&#8243; />
<KeyInfo xmlns=”http://www.w3.org/2000/09/xmldsig#”&gt;
<KeyName>Rsa Key</KeyName>
</KeyInfo>
<CipherData>
<CipherValue>fV3NsFhZR/l0/5nvioFfjwjhhauNUTR96fQOK3QeRTW05ERDAQrFGj9MBt5Jh7Ca4rIS2JZfOfNTjTxWiEp/tjk+9LXVyPKrJYMiNlYiUmZGfV/amPsLPmRm2pOEyKwJhJLN6NyZdht/xGrf1ClDKO6CG1ViA5pK5R8Db8X9ul4=</CipherValue>
</CipherData>
</EncryptedKey>
</KeyInfo>
<CipherData>        <CipherValue>eFvSbAzbVUzwa9Sl8V6t43kuwAcvmaPUjboSJ/oi+MMJXyqtqXS8dKSuxBy+E0rC8tUxxIfJppZNm+CCoKf9Rm39vW2flpgcsvm8ZNMekSf4r2GWYAvLw3vYvMBcbnFRqktlaM7cXia38+3KGN8skHzxioqrBgy2QQqqPWIPrmrCS440BRlXEck6XwAO9rZOERgM6+OtlRan4EuGoB0O4acJWbp51zWxkfzqxMb600BHkYzeIYkHH8GNvWo+LSQt6o+NYW+Q7sm/lLFY5hPp3pGTOygXPehT1b/3BWZM+1dJ5sh8sBXO+t5m7/Dzqt4nvMqArmdEUvQdhYAPauC3Uj9HjDFpHkbOjVEzohIvB0kJ1Wc3uP4VvE6CRMbAsrRiSNLDlT6OpXYVrArLk9c1bBA56nFXPMxLEpN1umRcCfaQY0qxKrZi/yJ8dKD/C/5Vo7o50f10jM9eUrt3/uS71bNJk5U9N7kO42tZZGXZMui51o6MWcYxSC7VQ3KdCpy6UacBnD8MYr7EHeZ591ATQds8dzcsXY7w6Lsg1pXLK74HqXMW/xDeLtBoWJxat9y+</CipherValue>
</CipherData>
</EncryptedData>
</connectionStrings>

I hope this helps.

Don’t Disrespect the Web.Config – Transformations

No matter how large of a project you are building nor how many lines of code you maintain, the most important file in your whole solution is likely the web.config file. It potentially contains connection strings, API keys, passwords, etc. If any of this information is incorrect you will likely see many problems in your application. Likewise, if a hacker were able to examine this file, it could mean disaster for your network. It is for these reasons that the web.config file must be treated with the utmost respect.

There are 3 things every public website should be doing with their web.config

  1. Use web.config transforms to keep track of development versus production settings
  2. Encrypt important configuration sections for security
  3. ELMAH – Error Logging Modules and Handlers

Each of the above topics will be covered in a separate post. As for today, we’ll discuss #1. Visual Studio 2010 introduced web.config transforms, which make it dead simple to maintain configuration information for multiple deployment environments.

The scenario:

Imagine the not-so-rare network setup of a website that is deployed on a production server, a test server, and is run locally by developers. In the old days, it was difficult to keep track of all the different environment-specific configuration options. Maybe you set the web.config correctly once for each environment and just never overrode it during a new publish. Maybe you created your own configuration text files that were dynamically linked into the application. Either way you had to spend extra time to solve this seemingly simple problem.

Thank goodness for modern IDEs.

Now it’s extremely easy to setup multiple configuration files in your Visual Studio solution, with these additional benefits:

  • Easy integration into source control
  • Each separate web.config file is tied to a Visual Studio build configuration
  • Easier integration into automated builds and deployments

Since the web.config transformation technology has been around for about 2 years now, I’ll try and introduce a new spin on it by demoing this with the Visual Studio 11 Beta.

How to set it up:

  1. Open your project in Visual Studio 11 Beta. Note: new projects in Visual Studio 2010 and 11 already have multiple config transformation files.
  2. Open the Build menu and navigate to the Build Configuration manager.
  3. Click the Active solution configuration drop down list and select New.
  4. Enter information for your test environmentbuild configurations
  5. Right click on the web.config file and select Add Config Transform. Note: If this option is not available, see this link.web.config transforms
  6. You will now see a web.config for each environment you support (Debug representing the developer’s local environment, Test representing Test, and Release representing production).

    config transform

  7. Next, we introduce the environment specific settings for your web.config files
    1. My web.config files typically have environment-specific information in the appSettings and ConnectionString sections:

    <appSettings>
            <add key=apiKey value=“83ABC029538FED091ACDD”/>
    </appSettings>
    <connectionStrings>

    <add name=DBConnectionString connectionString=Data Source=DBServer;Initial Catalog=DatabaseName;Persist Security Info=True;User ID=userName;Password=password providerName=System.Data.SqlClient/>

    </connectionStrings>

    1. Since the above is configured for my development environment, I usually leave my web.Debug.config blank
    2. In the Web.Test.config and the Web.Release.config I replace my appSettings and connectionStrings sections with environment specific values (see this website for detailed information on syntax)

    <configuration xmlns:xdt=http://schemas.microsoft.com/XML-Document-Transform>

    <appSettings xdt:Transform=Replace>
            <add key=apiKey value=B153439AB3DE8FF9CA9D0/>

    </appSettings>
    <connectionStrings xdt:Transform=Replace>

    <add
    name=DBConnectionString
    connectionString=Data Source=ProdDBServer;Initial Catalog=ProdDatabaseName;Persist Security Info=True;User ID=userName;Password=password
    providerName=System.Data.SqlClient/>

        </connectionStrings>

  8. Now just choose the build configuration and publish a project. It will automatically merge the correct web.config values based on the selected build configuration.

    asp.net publish

 

Beautiful. Your app has been deployed with the appropriate environment-specific web.config settings. In the next post, we will discuss how to encrypt secure information that is stored in the web.config file.

How I Enjoyed the Rock Paper Azure Competition

Rock Paper AzureIn mid-December, I saw an ad on StackOverflow.com and was immediately intrigued. “Rock, Paper, Azure!” was a contest run by Microsoft wherein programmers design bots to compete in a modified game of Rock, Paper, Scissors. The bots had to be hosted in Microsoft’s cloud computing platform, Azure, so you can easily see Microsoft’s motivation to give away some small prizes to influence developers into trying and (hopefully) adopting Azure.

Although I had plenty of things to keep me busy leading up to Christmas, the Rock, Paper, Azure marketing worked on me. I figured I could take 1 or 2 hours out of my time and write the best algorithm I could in that time. Besides, I would be entered into the grand-prize contest drawing just for competing with even the most simple of bots.

Bugs LanguageI was immediately reminded of a school project from an early Computer Science course at Ohio State. The contest back then pitted “bug bots” from teams of students in the course against each other. Each team started out with a handful of bugs on a large virtual checker board. A bug could “convert” another student’s bug by facing it and issuing the “bite” command. The bitten bug would then become a member of the “biting” bug’s army. The game continues until one team has converted all bugs. If I remember correctly, there were only a few possible commands:

  • “Detect” if an object (like a wall or another bug) was in front of it
  • “Move” forward 1 square
  • “Rotate” left or right
  • “Bite”

 

It may have evolved since then, but our bot did surprisingly well back then despite a very simple algorithm:

  1. If something in front, turn left, bite.
  2. Else, move forward, bite.
  3. Repeat Step 1.

 

I’ve often wondered what additional strategy I would write into my bot if given another opportunity in such a competition. Rock, Paper, Azure was the challenge I was looking for.

Microsoft’s version of “roshambo” came with a few twists, such as the introduction of the dynamite and water balloon moves. Check out all the details and rules here. I liked that it was a simple game but with competition against other developers’ bots came many options for creative strategy. Additionally, I was extremely impressed with how simple it was to build the basic bot.

Game Rule Highlights:

  • Bots compete each other throwing one of Rock, Paper, Scissors, Dynamite, or Water Balloon
  • Normal rules apply except that the Dynamite beats everything but Water Balloon and Water Balloon loses to everything but Dynamite
  • Each bot only gets to use Dynamite 100 times
  • First bot to win 1000 times wins the entire match
  • Ties carry over, so the next round could be worth more than 1 win (similar to “Skins” in golf)

 

It took me some iteration to come up with my eventual strategy, which turned out to be admittedly mediocre (98th place out of 162). I realized that my bot can keep track of the history of moves that it has made as well as the moves of my opponent. My plan was to try and detect if my opponent was falling into a sort of pattern. I was especially concerned about the end of the round when we both would be desperately throwing dynamite to close out the match. As you can see, my strategy only had a small amount of success.

Nonetheless, I thoroughly enjoyed my time creating and deploying my bot. I encourage Microsoft to search for more clever ways to get developers interested in learning and using their development platforms. In this contest, I got to expand my mind, learn more about Azure, and I even got a free t-shirt. Here’s to the next competition!

3 Silverlight Architecture Tips with Brad Himelstein

Recently, I dove back into XAML-based development to create some Silverlight plugins for a large web application. Most of my experience in XAML came from writing a WPF application from 2008 to 2009, so I had much to learn about the asynchronous data retrieval paradigm used in Silverlight. I had questions about UI Design Patterns, Service-Oriented Architecture (SOA), and how to reuse class libraries, so I asked Brad Himelstein, from CinCom, if I could pick his brain while we hit a few golf balls. Brad has continued working with Silverlight and WPF for a few years, since we worked together on the aforementioned WPF application.

Below is a summary of his tips as well as some quotes from our conversation.

 

Tip #1: Use Model View ViewModel (MVVM)

Me: “Do you ever use any sort of patterns for the UI?”

Brad: “MVVM

Me: “I am just now getting introduced to that. What are the benefits?”

Brad: “It is great, because you just setup the data context and do all your databinding, and it just magically works. Two-way binding works as well.”

 

Brad identified additional benefits during other parts of the conversation. MVVM’s primary purpose is to separate the concern of the behavior of an application from that of the user-interface design. It is a practical implementation that is cleaner to understand and easier to pass back and forth between designers and developers versus the default approach of placing logic in code-behind files. He provides sample MVVM source code here.

 

Tip #2 Extend Proxy Classes with Shared Custom Code

In my application, I use WCF RIA Services to query server objects and return them to the Silverlight client. This technique simplifies the process of retrieving service data and populating local objects because a local proxy object is created and populated automatically. In Brad’s words, “If the web service returns an object of class ‘Foo’, we don’t need to redefine ‘Foo’ in the Silverlight app.”

Unfortunately, I had a great deal of logic included with the retrieved classes that I also wanted to be able to use in the client. I asked Brad how to elegantly work around this limitation of Service-Oriented Architecture.

Me: “What I was hoping to get with RIA Services was, say I’ve got some class I’m returning from the service. I can load it with data easily but I have additional calculation properties on this class, such as FullName, which is just a property that returns FirstName and LastName concatenated. A local proxy object is created automatically for use in Silverlight, with data populated for FirstName and LastName, but the proxy does not retain calculation logic. Since I have a server-side object that has calculation properties defined, how can I share that code for use with the Silverlight client and the proxy object?”

Brad: “If you know you’re going to use FullName, then just create another string called FullName and set the property on the server so it is passed down to the client in the object, because it is not going to take those calculated properties and bring them down.”

Me: “But then that FullName would have a state when it returns.”

Brad: “Remember, you can extend everything. So on the client side, you can alternatively have your own Foo class, which can be used to extend the partial Foo class that is returned from the server.”

Brad’s suggestion was enough to help me come to a solution that fit my needs.

 

Partial Class Extensions

Let’s use the sample Class “User” to continue our example from above. In such a class, I can place fields of the User, such as FirstName and LastName in one file named User.cs. User class should be defined as a partial class, so we can combine definitions of the class from 2 separate files. Hence, in another file, named UserCalcs.cs, we should define calculated properties, such as FullName. At this point, the server behavior will work as it always would have if we had just implemented all properties and fields in the one file. However, the Silverlight client would not yet know about the FullName property.

 

Shared Files

An important limitation of Silverlight is that it can only reference projects that are compiled as Silverlight projects. In other words, Silverlight cannot natively reuse definitions from standard .NET class libraries. What it can do is recompile normal .cs files into a Silverlight assembly.

To do this, right-click on the Silverlight project in your Visual Studio solution and click Add Existing Item. Navigate to the file that contains the calculated properties, UserCalcs.cs. Click the file, and then click the down arrow next to the Add button. Choose Add As Link.

The Silverlight project will now compile this partial class definition. It will extend the proxy class as well as the server side class from the same file. Therefore, maintenance of those calculated properties can be shared between the client and the server side code from one file.

 

Tip #3 Learn HTML 5

Ok, Brad didn’t actually say this so bluntly, but it was a recurring theme of our discussion. Based on recent information about upcoming Windows (8) releases, Silverlight’s future is unclear. From what I have heard, Microsoft is committed to version 5 of Silverlight (current release is 4) but has made no guarantees beyond that. Windows 8 will have 2 versions of its Internet Explorer 10 browser: tablet and desktop. The tablet version will not allow plugins, like Flash and Silverlight. Some have summarized the decision by stating that from Microsoft’s standpoint, Silverlight is “no longer strategic.”

Still, the death of Silverlight development, especially in corporate environments, is distant. Here is a great Microsoft client decision workflow about the correct technology to leverage based on your needs. I believe it reinforces the idea that even after Windows 8 is released, new Silverlight applications will have their place.

Lastly, even though Windows 8 tablets will not allow Silverlight plugins to be loaded within its browsers, native Windows 8 tablet apps will be developed using WinRT. As Brad informs us, this gives us Silverlight developers hope, as WinRT is still “just C# and XAML.”

 

By

How to Query the Yahoo Fantasy Football API in .NET

The Coveted League TrophyAs technical co-commissioner of my Keeper Fantasy Football league, I perform a good deal of administrative duties during the offseason. I have to sort through all the player transactions from the season to determine which NFL players are eligible to be kept within our league rules. One year, at the end of the 2010 season, I waited too long to gather this data. I normally click through the Yahoo Website to see all the historical transaction data necessary. However, after a certain point in the year, Yahoo removes access to the year’s information.

Luckily, in late 2009, Yahoo opened up their Fantasy Football API. Therefore, I was able to implement a .NET solution to retrieve the data I needed. Below are the steps I took.

The goal of our solution will be to create a simple ASP.NET Web Forms application that authenticates to Yahoo and then enables us to query the API.

Perhaps the most difficult hurdle to overcome is authentication. Yahoo uses a web standard called OAuth, which is a “simple, secure, and quick way to publish and access protected data”. You can find out more about it here, as I do not plan to delve into the low-level details of this protocol. The difficulty I found was that there are not many examples of .NET applications using this technology on the web.

To help us authenticate, we will leverage a .NET library called DevDefined OAuth.

  • Create a new Web Forms solution and add a reference to this project or compiled dll.

In our next step, we will create a simple button (named AuthenticateButton) in the Default.aspx page to kick off the authentication process.

  • Create a click event handler for your button. Enter the following code in the code-behind:

string requestUrl = https://api.login.yahoo.com/oauth/v2/get_request_token&#8221;;
        string userAuthorizeUrl = https://api.login.yahoo.com/oauth/v2/request_auth&#8221;;
        string accessUrl = https://api.login.yahoo.com/oauth/v2/get_token&#8221;;
        string callBackUrl = http://domain.com/Query.aspx&#8221;;

         protected void AuthenticateButton_Click(object sender, EventArgs e)
        {
            var consumerContext = new OAuthConsumerContext
            {
                ConsumerKey = “[provided by yahoo]”,
                SignatureMethod = SignatureMethod.HmacSha1,
                ConsumerSecret = “[provided by yahoo]”
            };

             var session = new OAuthSession(consumerContext, requestUrl, userAuthorizeUrl, accessUrl, callBackUrl);

             // get a request token from the provider
            IToken requestToken = session.GetRequestToken();

            // generate a user authorize url for this token (which you can use in a redirect from the current site)
            string authorizationLink = session.GetUserAuthorizationUrlForToken(requestToken, callBackUrl);

             Session[“oAuthSession”] = session;
            Session[“oAuthToken”] = requestToken;

             Response.Redirect(authorizationLink);
        }

 

To obtain your ConsumerKey and ConsumerSecret strings to place into the above code, go Yahoo’s Developer Projects Site and create a project. When creating your project, enter the Application URL (e.g. http://domain.com) and App Domain (e.g. domain.com) of the deployed location of your web site. Also, make sure to enable access to the Yahoo Fantasy Football API.

Take a look at the value in the callBackUrl string above. This needs to be edited to be an address on your web site. As part of the authentication process, Yahoo calls back to a URL on your site and therefore requires your site to be accessible from the public web.

  • Add a new web page to your project named “Query.aspx”
  • Place the below code in your Page_Load event handler

        protected void Page_Load(object sender, EventArgs e)
        {
            OAuthSession session = (OAuthSession)Session[“oAuthSession”];
            IToken requestToken = (IToken)Session[“oAuthToken”];

             // exchange a request token for an access token
            string oauth_verifier = Request.QueryString[“oauth_verifier”];
            if (!String.IsNullOrEmpty(oauth_verifier))
            {
                IToken accessToken = session.ExchangeRequestTokenForAccessToken(requestToken, oauth_verifier);
                Session[“oAuthSession”] = session;
            }
        }

 

Now that we are finished implementing the authentication code we can write the query submission logic.

  • Add a textbox (named QueryTextBox), a label (named ResultsLabel), and a button (named QueryButton) to the Query.aspx page
  • Add a click event handler for your button. Enter the following code in the code-behind:

        protected void QueryButton_Click(object sender, EventArgs e)
        {
            string query = QueryTextBox.Text;
            IConsumerRequest responseText = ((OAuthSession)Session[“oAuthSession”]).Request().Get().ForUrl(http://fantasysports.yahooapis.com/fantasy/v2/&#8221; + query);
            ResultsLabel.Text = responseText.ToString();
        }

 

You are now ready to query the Yahoo Fantasy Football API. Here is how you use it.

  1. Load the page by navigating to http://domain.com (replace “domain” with your website domain name).
  2. Click the Authenticate button

  1. Yahoo’s site will guide you through the steps to login to your Fantasy Football account and redirect to your site when finished.
  2. Enter an API Query into the text box and click “Submit Query.”

 

There are definitely improvements that can be made to our sample application. For instance, the results output from our query are not easily readable, but they can be simply output to an xml document or some other format for reading.

The amount of data that can be extracted from the API is huge and can be leveraged for some very creative purposes. For more information about the query syntax as well as available data, see the Fantasy Sports API Documentation.

You can download the code for the sample application.

 

Lastly, below are some resourceful links:

http://github.com/buildmaster/oauth-mvc.net

http://oauth.net/code/

http://oauth.googlecode.com/svn/code/csharp/

http://blog.techcle.com/2010/03/20/simple-oauth-integration-for-twitter-in-asp-net-mvc/

http://www.codeproject.com/KB/cs/Delicious-OAuth-API.aspx

http://code.google.com/p/devdefined-tools/w/list

http://developer.yahoo.com/fantasysports/

http://developer.yahoo.net/forum/index.php?showforum=122

 

5 Things I Hate about my Favorite Programming Language: C#

In episode 73 of the StackOverflow podcast , Jeff Atwood mentions one of his favorite questions to ask developers, “What are 5 things that you hate about your favorite programming language?”

It got me thinking. I definitely like some of the language features of C#, especially when developing within Visual Studio. Some of my favorites are Generics, Intellisense, and Short-Circuited conditionals. However, there are other pains that I encounter repeatedly when it comes to language syntax.

 

  1. In C#, there is the nifty syntax of following a variable with “??” allowing the developer to specify a replacement value in the case of the variable being null. In theory this is a great feature. However, I never end up getting to use this and I imagine that is the case for most developers. I find I usually end up using the “?” and “:” syntax because I typically want to use a member of the null object like “user.Identity.ToString()” but I have to check if “user” is null or I will get an object reference error.
  2. Why do I always have to lookup how to format a DateTime object in code without the minutes?
  3. If “ToString()” is a member of every object, why should I have to explicitly call it when setting the value of a string variable from another non-string variable? As an example, why can’t a string value be inferred from my int variable by just calling “ToString()” when there would otherwise be a type issue?


    int i = 0;

    string j = String.Empty;

    j = i;

  4. The compiler considers the use of “=” valid within an if condition. This can be confusing and often causes accidental issues where “==” is the intended code. As a solution, I use a coding standard of typing the constant value to be compared first because it cannot be assigned to (i.e. “if (0 == count)”).
  5. Nullable types seem like a great idea but they are not always intuitive. In the example below, the code seems like it should work. However, I receive a compiler error on the second line.


    int? i = null;

    i = (sender != null) ? sender.GetHashCode() : null;

 

Most of the above complaints were compiled off the top of my head. If you have found a better way to leverage the C# language so that you do not run into these issues, please share them with me in the comments. Perhaps it is my mis-use of the language features that has caused my agony.

I Heart Karnaugh Maps

Have you ever found yourself writing a long Boolean condition in your code like the line below?


if ((policy.Type == PolicyType.AutoInsurance && policy.PolicyHolder.PriorAccidents == 0) || (policyPaidInFull && policy.Type == PolicyType.AutoInsurance || policy.IsPremium) || (policy.Type == PolicyType.AutoInsurance && policy.PolicyHolder.PriorAccidents == 0 && policy.IsPremium))

 

I Heart Karnaugh MapsPerhaps that line of code above is the first and easiest way you thought about all the conditions that have to occur in your application’s business logic. You write that line of code, test it in many different scenarios and it works so you think you have done a good job. Well, there are ways to improve upon that line of code by removing logically equivalent Boolean expressions, not to mention some style improvements that might make it more understandable.

We owe gratitude to our dear friends the Electrical Engineers for developing a clever tool, named Karnaugh Maps, to help with this dilemma, usually for cases of no more than 6 variable conditions. Karnaugh Maps (pronounced “car-no” and often simply called “K-Maps”) are a system for reducing Boolean expressions into a more simplistic form. They originated from the need to reduce electrical wiring gates, or circuit minimization, but they are still useful for the high-level software developer.

 

Benefits of reduced Boolean expressions

  1. Increased program performance
  2. Increased readability of code
  3. Less code results in easier to change code

Reducing our example expression

The 1st step is to let letters represent each of the conditions in our expression. In our case:

  • a :     policy.Type == PolicyType.AutoInsurance
  • b :     policyPaidInFull
  • c :     policy.PolicyHolder.PriorAccidents == 0
  • d :     policy.IsPremium

Then, draw a graphical square like the one pictured below. This represents each possible combination of our Boolean conditions. The boxes are blank because we have not yet entered what Boolean results we want in our resultant expression.

Blank Karnaugh Map

Let’s take the first, simplified condition in parentheses, if (a && c), and put it into the map. The result would look like the below image, because we only need to fill in the boxes where a and c are both 1.

Sample Karnaugh Map

Following this example, we can use the entire Boolean expression to fill out the whole map. The completed Karnaugh Map is pictured below.

Completed Karnaugh Map

Now circle any square or straight line of boxes since they correspond to an expression that differs by only two bits.

Circled Karnaugh Map

Because of the way we have arranged our variables around the outside of the map, we can eliminate variables based on boxes filled with 1s being adjacent to each other. In our example, the vertical “circle” exhibits a scenario where the expression should always be true as long as both the a bit and the b bit are 1, hence the condition ( a && b ). Likewise, the square “circle” exhibits true cases whenever the c bit is 1 and the a bit is 1. Because the b bit and d bit are true no matter if their values are 0 or 1, they can be deleted from the resulting simplified expression, which would be || ( a && c ). We use a similar rule to find the final d condition.

In one sentence, this rule can be summarized as “the circled boxes can be grouped together and the two variables that differ can be discarded.”

The resulting Boolean expression is:

if ( ( a && b ) || ( d ) || ( a && c ) )

which, using Boolean algebra, can be further reduced to:

if ( ( a && ( b || c ) ) || ( d ) )

We were not able to completely remove any variables, but we did simplify the expression quite a bit. The original Boolean conditions can be substituted for our letter variables, and we can rewrite the original expression as below:


bool policyIsAuto = ( PolicyType.AutoInsurance == policy.Type );
bool zeroPriorAccidents = ( 0 == policy.PolicyHolder.PriorAccidents );
if ( policyIsAuto && ( policyPaidInFull || zeroPriorAccidents ) || policy.IsPremium )

Seems easy, right? It is. And I am sorry if my steps went too fast for you. My intention is not to teach how to use Karnaugh Maps for all circumstances but instead to show you how easy and useful it is to simplify your Boolean logic.

For help with different scenarios, find the book Bebop to the Boolean Boogie – an Unconventional Guide to Electronics at your local library, check out this Wikipedia link, or you can even download software to perform the rules for you.

Happy Karnaugh Mapping!

In the mean time, I hope I was able to show you how fun and easy using a system like this can be. Feel free to post questions in the comments.

 

– Karnaugh Map Images taken from Bebop to the Boolean Boogie – an Unconventional Guide to Electronics

– Digital Logic image created by Garrett Crawford