adrift in the sea of experience

Tuesday, November 18, 2008

Executing visual studio unit tests without installing visual studio

UPDATE: I've written a new post for Visual Studio 2010.

The professional edition of visual studio 2008 has a nicely integrated unit testing framework. I suppose visual studio team server will run these for you whenever somebody checks in a source code change. Unfortunately, we don't have team server or the team edition of visual studio at work.

This lead me to look for a way to execute these unit tests on our build server. "build server" may be a bit of an euphemism; it is little more than a virtual machine with a collection of python and batch scripts and it doesn't have visual studio installed. In case you are wondering, compiling C# projects and solutions from the command line with msbuild works fine without installing visual studio. I wanted a similar solution for the unit tests.

It turns out that there is a command line utility called mstest.exe that can be used to run these tests. This utility comes with visual studio. It is not straightforward to get it working without installing visual studio. Here's what I did to get mstest.exe running on our build server:

1) Created a \tools\mstest folder on the buildserver
2) Copied %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe
3) Copied these files from %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies

Microsoft.VisualStudio.QualityTools.AgentObject.dll
Microsoft.VisualStudio.QualityTools.CommandLine.dll
Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.ControllerObject.dll
Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
Microsoft.VisualStudio.QualityTools.TMI.dll


4) Copied these files from c:\windows\assembly\gac_msil subfolders. This requires you to use the command line as windows explorer hides the actual files there.

Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Resource.dll


5) Added certain registry keys to the build server registry. You can save these in a .reg file and then simply double click it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\HostAdapters\VS IDE\SupportedTestTypes]
"{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes]
@="TestTypes"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.SUnitTestService, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#212"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{00000000-0000-0000-0000-000000000000}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestTip, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\Extensions]
".dll"=dword:000000d7
".exe"=dword:000000d7

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\SupportedHostAdapters]
"Smart Device"="Smart Device Host Adapter"
"ASP.NET"="ASP.NET Host Adapter"
"VS IDE"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.SOrderedTestService, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#313"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{700218d8-f6f1-4ec3-be76-a35f72260503}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.AutoSuiteTip, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}\Extensions]
".orderedtest"=dword:0000013c


After that, you can invoke mstest.exe on the command line like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"


mstest.exe will return an error code if any test fails.

I have not figured out how to get things working without the /noisolation switch. I believe that any problems caused by this switch are rare. In any case it will only cause problems if there are serious issues with the code under test which will need to be fixed anyway.

Wednesday, May 21, 2008

Lessons from IT projects: allowing NULL foreign keys in a database can be tricky

My previous "lesson learned" was a bit general, and perhaps not very useful. Migrating old data is hard, but if the problem wasn't hard to fix we wouldn't be payed to fix it now would we? Today I'll focus on a much more simple technical dilemma: whether to allow database foreign keys to be NULL.

Consider a database table of contacts, a classic example. Each contact has a name, telephone number, email, country_id, etc. The country_id field for the contact is actually a foreign key pointing to a country record in the country table.

Consider now that often we only have a few pieces of contact information, e.g. only a name and email address. If we don't know the country, we can't set it. It may seem natural to declare the country_id column to allow for NULL values.

It's a possibility, but it has consquences. What kind of consequences?

Suppose we'd like to pull all contact data from the database and show it on the screen or in a report. We also want the country data in the same view. The natural query for this is


SELECT * FROM Contact, Country WHERE Contact.country_id = Country.id


Clean and simple. And unfortunately, broken. Your report will silently omit all contacts where country_id is NULL! Needless to say, this can be a serious bug if you are generating a billing/bookkeeping report.

To fix this, you'd have to use a "LEFT JOIN" operation:


SELECT * FROM Contact LEFT JOIN Country ON Contact.country_id = Country.id


Or you could just forbid country_id from ever being NULL like this:

CREATE TABLE [dbo].[Contact](
[id] [int] IDENTITY(1,1) NOT NULL,
[name] [nvarchar](999),
[address] [nvarchar](999),
[CNT_CountryID] [int] NOT NULL,
... more create table goodness ...

Then of course, you also need to put a special "UNKNOWN" country in your Country table. This solution has the "magic value smell" a bit, but after debugging a few problems caused by NULL foreign keys, I very much prefer to use a special unknown country over NULL. It also has the advantage that your code will need to deal less with NULL values, which can be annoying in itself. Come to think of it, those name and address fields should also be NOT NULL, just use empty strings when you don't know the values!

Tuesday, May 20, 2008

Lessons from IT projects: migrating existing data to a new system

I'm currently working on a redesign/reimplementation/integration of an administrative system that currently exists as a mix of 3 databases (with semantically vague and manually maintained links between them), two applications and lord knows how many excel spreadsheets. The goal is to have 1 database and 1 application, less room for mistakes, and drastically reduced administrative overhead.

As I approach the deadline for this project, I often find myself thinking "If only I had known that ..." or "I should remember X next time".

A first observation: Migrating old data to a new system is harder than you think. It forces you to follow stupid constraints for the new system, because otherwise the old data can't be expressed in the new system. Imagine trying to build a house in the middle of the forest without being allowed to cut down certain trees. You'll end up with alot of extra walls to build around the forbidden trees, and your shiny new house will start to look like a maze.

If at all possible, go for a green-field scenario where the data can be left behind or is migrated gradually/manually by the customer. After all, it's the customer who knows about the old data and business rules. If you absolutely need to migrate the existing data, don't underestimate the effort of having to map out the existing organically evolved "solution". That may be a project in it's own right.

Sunday, February 24, 2008

Logging all your chats in one place

I currently use used to use pidgin for chatting, mainly because it is cross-platform and I liked using the same application on windows and linux. Another advantage of pidgin was that it is designed to support almost all the chat protocols out there. I can be online on MSN and google talk at the same time with the same application.



One thing has been bugging me for a while now though, and that's the fact that my pidgin chat logs are stored locally. "Locally" turns out to mean "spread out over several machines and user accounts", without any hope of making them searchable or properly backed up. This is not a good thing. Fortunately this is only a problem for MSN chats. It doesn't matter for my google talk account, as the google talk server will log everything for me "in the cloud" and even make it searchable from the gmail web interface.



Is there anyway to get this advantage for other chat accounts also? Well actually, it turns out that there is. Google talk is just an jabber/XMPP implementation, and google has enabled "federation" with other jabber servers. That means that anyone can run their own jabber server, create users for it, and chat with google chat users. The jabber spec also includes the concept of "transports", which are basically gateway services running on the jabber service which enable you to talk to other networks like MSN. All you need to do is register an MSN transport with your google talk account, do all your MSN chatting via your google account, and voila: all your MSN chats are stored at google.

Here's how to do it:
1) Google talk or pidgin doesn't enable you to add transports. Download and install psi instead, another jabber client.
2) Psi will prompt you to add a jabber account when you first start it. Use your gmail address as a jabber-id. Set the status to "online", you should be connected to google talk now.
3) Right click on gmail.com in psi and click "service discovery". Psi will automatically look for services at "gmail.com" and show an error message because there aren't any.
4) Enter another jabber server in the service discovery window, one which runs an MSN transport (e.g. jabber.belnet.be) or some other transport for your favorite chat network.
5) Locate the MSN transport in the list of services, right-click it and click "register". This will prompt you for your MSN credentials.

6) Add/Authorize the transport contact which now appears in your main psi window (the "contact roster"). Double click it and allow it to subscribe to your presence.
7) All your MSN contacts will now also appear, each one needs to be accepted for addition in your contact list. (It may be faster to close psi at this point, log in with pidgin and accept the contacts from there.)
8) Say something to one of your MSN buddies
9) Go to gmail.com, log in, click "chats" or search for what you just said. It's all there! :) All your MSN chats are now logged by google, as long as you use your name@gmail.com jabber account for chatting.

Thursday, February 21, 2008

Finding debian packages and listing the files installed by them



The great thing about GNU/linux distributions is that they come with zillions of software packages, downloadable over the internet from free software repositories. Say you notice that your firefox browser doesn't support java applets. In debian, you're just a

apt-cache search java mozilla

away from finding a package providing just what you need. One of the results returned on my machine by the above command was "sun-java6-plugin". Just do

apt-get install sun-java6-plugin

and you're done! No need to worry about dependencies (like, ehm, java!), those are also downloaded and installed as needed.

Using an installed package

When you install an application "foo", typically it will be in a package named "foo" and will be launchable by typing "foo" in the console. In the case of command line tools, typing "man foo" will bring up one of those fascinating manual pages. In the case of GUI tools, they should be automagically added to your start menu in your favorite window manager. So far so good.

Unfortunately, a minority of the packages are a bit evil in this aspect. The "bittorrent" package for example provides the original bittorrent implementation as a set of command line tools. You can install it, but you'll find that no "bittorrent" command becomes available, nor will "man bittorrent" give any results. "apt-cache show bittorrent" doesn't provide much hints either. Where the hell are my command line tools?!

There is a solution! It took me a while to find, but it is to do a

dpkg -S bittorrent|grep bin

which will list all the invokable binaries installed by the bittorrent package. Turns out that the tool is invokable as btdownloadcurses.bittorrent (thank you tab completion!).

Getting started with a new programming library

This is also very handy for other types of packages for which it may not be obvious how to start using them, like programming libraries. Suppose you want to write a little script to upload your pictures to google's picasaweb service. A quick

apt-cache search picasa

will yield the "python-gdata" package. Once installed,

dpkg -S python-gdata

will help you locate all the new python goodness at your disposal. In this case, it turns out that alot of sample scripts are installed at /usr/share/doc/python-gdata to help you get started.

Monday, February 4, 2008

Recording phone calls on a S60 phone. Part 3: getting rid of the beep

OK so recording phone calls turned out to be relatively easy, but what about that annoying beep inserted into recorded phone conversations? This beep comes after the first second of the conversation, and every 15 seconds after that. Turns out that there is a way around this: the trick to avoid the beep is to not record longer than 1 second.

That's a pretty short conversation, you say?

Not necessarily. You could, for example, stop the recording before the first second completes, then restart it immediately, etc... Unfortunately the speed at which code can be executed is limited, and this technique results in unacceptable gaps in the recording, especially if we use python. There is a commercial application called Total Recall which appears to use this technique. It worked for my N95 if I set the recording quality to "low", but like my python script, the recording gaps are unacceptable.

Is it possible to do better? Yes! If you start a new recording before stopping the previous one, the beep still doesn't surface, and there will be no lost audio. The following function does exactly that:


def start_recording(filename):
global recording
counter = 0
recording = True
sound_file = audio.Sound.open(filename+'_part'+str(counter)+'.wav')
sound_file.record()
start = True
while recording:
if start:
start = False
else:
e32.ao_sleep(.6)
counter = counter + 1
next_sound_file = audio.Sound.open(filename+'_part'+str(counter)+'.wav')
next_sound_file.record()
e32.ao_sleep(.3)
sound_file.stop()
sound_file.close()
sound_file = next_sound_file
print "stopping recording"
sound_file.stop()
sound_file.close()
concatenate_wavs(filename, counter)


This function will create a recording in fragments of less than 1 second, which overlap about .3 seconds. In practice, it turns out that the overlap is typically a bit less than that. Now we could post-process these audio fragments on a PC to stitch them together, but part of the usefullness of recording phone conversations is that you can immediately replay them on your phone. This implies the need to do the post-processing on the phone itself.

This is were the python wave module comes in. This module provides all the functionality you need to read and write valid .wav files, but unfortunately it is not included in the PyS60 implementation. Not one to be easily discouraged, I downloaded the official python 2.2 sources. This is the (old!) version upon which PyS60 is based. Unpack the source archive, copy the wave.py and chunk.py modules to the phone, and yes, it works!

Simply concatenating the wave samples will result in a little stutter in the recording though. The following function attempts to detect the overlap between fragments to fix this:


def concatenate_wavs(filename, max):
print "Processing "+str(max+1)+" audio chunks:"
merged = wave.open(filename+'.wav','w')
previousframes = None
for i in range(0,max+1):
partname = filename+'_part'+str(i)+'.wav'
part = wave.open(partname,'r')
frames = part.readframes(part.getnframes())
part.close()
os.remove(partname)
if i==0:
merged.setparams(part.getparams())
overlap=0
else:
overlap=frames.find(previousframes[-8:])+8
if overlap==7:
overlap=0
print overlap
previousframes=frames
merged.writeframes(frames[overlap:])

merged.close()
print "done!"


And there you go, a simple python script for recording phone conversations on a S60 3rd edition phone without beeps :) For some reason the overlap detection does not always work, but the casual stutter is not yet annoying enough to warrant further investigation. More important is the fact that this script still needs alot of work before it can be declared suitable for public consumption. It should get a proper GUI, be packaged as a standalone application which can be easily installed, and it should be started automatically whenever the phone is booted.

But I have not yet figured out all that stuff, so that's for another post... :)

Friday, January 25, 2008

Recording phone calls on a S60 phone. Part 2: working prototype

<voice type="farnsworth">Good news everyone!</voice>

I messed around with the PyS60 python interpreter for my S60 phone, and managed to record phone calls! The first step is to install PyS60 as described on the PyS60 wiki. No further tools are needed, the python runtime and shell on your phone are sufficient. Next, we need a script which listens for phone calls being connected, either incoming or outgoing, and which starts and stops recording those calls:

import appuifw
import e32
import telephone
import audio
import time

# Must be either None, or a sound file which is recording.
sound_file = None

# Directory used to save call recordings. Must exist.
# Filename will be appended to this path.
recordings_folder = "e:\\CallRecordings\\"

def cb_calling(args):

global sound_file
state = args[0]

# if transition to connected state and not recording, start recording
if (state == telephone.EStatusConnected) and (sound_file == None):
number = args[1]
if number == None:
number = "None"
filename = recordings_folder + time.strftime("%Y%m%d_%H%M_%S_") + number + '.wav'
#filename = recordings_folder + 'test.wav'
print filename
sound_file = audio.Sound.open(filename)
sound_file.record()

# if transition to non-connected state and recording, stop recording
elif (state != telephone.EStatusConnected) and (sound_file != None):
sound_file.stop()
sound_file.close()
sound_file = None


def quit():
app_lock.signal()

appuifw.app.exit_key_handler= quit
app_lock=e32.Ao_lock()

telephone.call_state(cb_calling)
telephone.incoming_call()

app_lock.wait()

The body of the script only sets the exit key handler, registers a callback function for detecting telephone call state changes, and starts waiting for a signal on a lock object. After that, the main thread will not do any more work. It was surprising to discover that real multi-threading is central to developing in Pys60. I was expecting to see a "message queue with single threaded message pump" kind of model, like most PC application frameworks.

Copying this python script to a \Python directory enables you to run it on the phone with the python shell application. Once running, it will listen for phone calls and record them to the recordings_folder.

Unfortunately, there is an annoyance: apparently nokia feels obligated to insert a beeping sound into the call every 5 seconds if it is being recorded. The beeping is not present in the recording though. I did not find any more information on this "feature", other than that it appears to be built into the phone. I guess nokia is trying to legally cover their ass here. I researched Belgian law regulating phone call recording, but did not find any obligation to inform participants that you are recording them. If you are not recording with intent to deceive or harm and are participating in the call yourself, then it's OK to record without warning. (Disclaimer: I am not a lawyer)

Can we get rid of these annoying beeps? Is there a way to automatically start the call recorder code on phone boot? Questions, questions...

To be continued!

Wednesday, January 23, 2008

Recording phone calls on a S60 phone.

I love gmail. I haven't needed to delete email since I started using it. And the search feature is awesome. I wish my entire life could be archived and searched just as easily. And why not? For example, everything I hear could in principle be recorded to files and archived. Add a dash of speech-to-text and I could have a searchable history of all my conversations.

Right now however, I only archive email and the occasional cameraphone snapshot or video (and chat logs and text messages, but I should archive them better). In pursuit of the goal "log everything", I'm going to try and add something to that list: phone conversations.

My nokia smartphone is build on the S60v3 symbian platform. There exists a python interpreter for that platform, and it comes with modules for doing a lot of stuff. According to the documentation, you can run code in response to incoming calls. You can also record audio. This should be all I need to start logging my phone calls.

Yay, we've got ourselves a project :)