Some notes for myself - connecting to Nike+

I'm thinking at the moment about extending my Hollybar GPS toy to allow it to talk with Nike+ devices and/or with the Nike+ website.

I've not done any real work on this so far, but here are some of the web hits which look useful:

• C# to Nike+ API discussion - http://www.wroolie.co.uk/?p=13
• Some simple XML - an example of the public run API - no idea who this user is!
• ear fung us - a wordpress plugin
• a php wrapper for the api - http://lerdorf.com/php/nikeplus.phps
• this site seems to be based on n+ - http://www.workoutdump.com/
  
• yet more discussions and examples at http://www.shokk.com/blog/articles/2007/01/10/nikeplus-api/ , and more besides
  

Just saving these items here in my stream of consciousness - sorry to anyone who has to read it! (plus I need to remember http://coursemapper.com/ for Garmin stuff too)

A fix for the Azure TableStorageMemberProvider

I don't know if this is already known about or solved elsewhere... I couldn't spot it anywhere... but I'll post my fix here anyway - it's pretty trivial.

Problem: in my Azure I tried to implement a simple verification email scheme for user registration using a scheme similar to that described on http://www.aspcode.net/Requiring-email-verification-for-new-accounts.aspx

To do this I had to use the ProviderUserKey for each user.

However, when I looked at this, each user seemed to be coming back with a zero (empty) Guid.

Fix: Looking in the Azure supplied TableStorageMemberProvider, it seems that a providerKey is generated but is never actually stored in the table storage.

To fix this... look in

        public override MembershipUser CreateUser(string username, string password, string email, string passwordQuestion,
                                               string passwordAnswer, bool isApproved, object providerUserKey,
                                               out MembershipCreateStatus status)

then look down to

             if (providerUserKey == null)
             {
                 providerUserKey = Guid.NewGuid();
             }
             newUser.Password = pass;
             newUser.PasswordSalt = salt;
             newUser.Email = (email == null) ? string.Empty : email;

and add a line

             if (providerUserKey == null)
             {
                 providerUserKey = Guid.NewGuid();
             }
#warning Fix implemented for zero'd providerUserKey
               newUser.UserId = (Guid)providerUserKey;
              newUser.Password = pass;
             newUser.PasswordSalt = salt;
             newUser.Email = (email == null) ? string.Empty : email;

That should work - it does for me :)

While working on Azure - some ASP.NET stuff covered

So my ASP is a little rusty...

But working on my first Azure website - www.stacka.com - has helped me oil out quite a bit of rustiness.

Among the items covered (so far):

• General ASP.NET structure - including master pages, pages, server controls, validation, viewstate, code behind, sending emails, using web.config, writing custom HTTPModules ...
• Some specific ASP.NET nasties:
• IIS7 integration - spent far too long working out why my HTTPModule wasn't being called in IIS7 - how was I to know all my web.config examples were IIS6 ones?
  
• URL rewriting - this MSDN article was especially useful as was Scott Gu's blog - the post back issue took me a day to code around... but eventually it was solved using 2 url rewrites - the first in BeginRequest, the second in PostMapRequestHandler
  
• AJAX general things - including some basic tutorials on things like UpdatePanels from asp.net
  
• Using the AJAX Control Toolkit - kudos to all who've contributed to this excellent set of tools.
  
• Integrating Google Custom Search - really this should have been easy - but a clash of forms meant some very odd errors appeared in my application (bizarrely I was seeing both Ajax and normal submit buttons work in Firefox, but break in IE) - eventually on the way to solving this using Scott Mitchell's notes on dotnetslackers.
• CSS integration - really, I've given up on this - it's just too hard integrating CSS and using things like the Login control... I'll try again later (maybe I'll use ASP.Net MVC and then I'll not use the standard login code?)
• Adding email verification to an ASP.Net user registration - easy - see this link on aspcode.net.
  
• Some specific Azure issues/challenges:
• How to live in a data store without JOINs
• How to keep data consistent without Transactions
• How to effectively use Partition and Row Keys as data lookup mechanisms
• How to implement aggregate methods (counts and averages) in the Azure Table storage
• What to do when optimistic locking is too optimistic (still working on this one!)
  
• How to put larger tasks on background threads (worker roles)
• Some bug fixes in the supplied samples for (for example) TableStorageMembershipProvider - really these samples are a godsend - I'm so glad that Microsoft supplied these!
  

Will post more details when I get my full invitation token from Microsoft - at that stage hopefully I will be able to upload my webapp and show it off a little...

An awful name for a very cute coding toy

Went to DDD at Msoft today - one thing that caught my attention was one catchy little number called "Inversion of control" - that's a really dire name, but underneath it's quite cute.

Basically:

• you take a system like a NetworkConnector which is built using tightly coupled objects - e.g. it references a NetworkMonitor, a Logger and a WirelessAdapter
• you rewrite it so that the NetworkConnector is an INetworkConnector, and so it uses an INetworkMonitor, an ILogger and an IWirelessAdapter - you then also add a simple constructor so that an external builder can pass each of these references in,
• then at runtime you can use an "Inversion of control container" object to completely wire your system together. There are a few implementations around - with one from "the Castle project" being the main one people seem to be using. Basically what happens is in your system construction you write code that looks a bit like:

Container theContainer = new Container();
theContainer.Register(INetworkConnector, typeOf(NetworkConnector));
theContainer.Register(ILogger, typeOf(Logger));
// etc ...

then at any time later you can ask theContainer for an instance of some interface - e.g. an INetworkConnector - and theContainer will then create it and it's dependent objects and wire them together... (to do this it uses a lot of reflection under the covers)

I know it may not sound that exciting, but what it means is that it's really quite easy to soft-code the relationships between classes - decoupling the classes and so making them easier to plug-in - especially useful for things like test harnesses - especially when used with Mock objects which seem to have come a long way since NMock (RhinoMock running inside TestDriven.net looked particularly neat).

Visual Studio 2008 SP1 Vista Ultimate Crash - Choose Items.... dialog

It seems this is a common callingrash....

I lived with it for a while.

But have finally found a good workaround - manually install the Power Commands for VS2008 (I think these might have been installed as part of SP1 but reinstalling them certainly does no harm and a lot of good!)

More info and other solutions at http://www.imaginativeuniversal.com/VisualStudio2008SP1ToolboxCrash.aspx

Azure data challenges

My first app on azure - stacka - is progressing quite well.

However, I keep hitting problems... especially when I try to do things which are very simple and straightforward if I was working with conventional SQL database - but they often prove somewhat more tricky in the distributed Azure storage.

Here's an example.

For stacka, I've got a StackTable structure - within this each StackRow is indexed by:

• PartitionKey - userid
• RowKey - a creation tickcount based number - similar to that used by smarx in his ongoing blogger worked example
  

My thinking behind this was that the partitioning would make it easy to search by user (which is a common task to do) which the row key would ensure that results were returned in time order.

However.... what I hadn't understood was that the sort order returned by the azure table storage is not rowkey - but rather is the tuple (partitionkey, rowkey) with normal lexicographical (alphabetical) ordering defined.

And (unlike in SQL) what I can't now do is reorder the stacks easily in the query - i.e. I cannot simple add a global "" to the query

So....... now I'm looking at:

• possibly rewriting this indexing, so that the Stacks are not stored with any preference for userid, but so that time based returning is still preserved. This will make it quicker to get the most recent N entities - but will make it slower to get all Stacks associated with one particular user.
  
• possibly adding an additional (manually maintained) table which stores the most recent StackRows with a time based ordering. Because this table doesn't need to store everything - just the top N entities - it doesn't need to be particularly large.
  

There are definitely some interesting patterns that will arise while coding with Azure....

Azure Invitiation code received :)

Just in case anyone else is waiting, it looks like the invitation lag on Azure at present is about 10 days - I've just got mine - so now I can start publishing Cloud Apps to Microsoft (US) for real.

This is what I just received:

"

Greetings from the Azure Services Platform Team!

Thank you for applying for a Microsoft® .NET Services and Microsoft® SQL Services invitation code on Microsoft Connect Website. (http://go.microsoft.com/fwlink/?LinkID=129453). Your invitation code for Microsoft .NET Services and Microsoft SQL Services will be HIDDEN-FROM-VIEW. Please note that this invitation code is NOT activated yet and cannot be used to create a solution at the portal at this moment. We will send you another email to this account once your invitation code is activated, but we will not send you your invitation code again. Please keep this email in a safe place.

Sincerely,
Jenny Lo
Azure Services Platform Team

"

25th November - Update to this post - what this actually was... was only an invite to the .NET and SQL bits of the Azure CTP preview - so I still can't upload Azure applications :( - most upsetting! Will let you know when the real invite arrives (please let it be soon!)

Outing to cloudcamp

Went along to London cloudcamp last night - http://www.cloudcamp.com.

Very interesting to see and hear a few of the cloud companies - big players like Sun, Amazon, Microsoft, VMWare, ... and smaller companies including http://www.zimory.com, http://www.flexiscale.com, http://www.witsbits.com and many many more besides.

It's clear from listening to a lot of the presentations and comments that this is still really early days for clouds.

However, it's also clear that people have been working in cloud tech for a significant length of time already - this is early tech but it's tech that can and is being used - with Amazon's AWS still the big leader.

I did find one thing particularly interesting - there seemed to be a lot of discussion about green-ness and about the environmental impact of clouds. I'm sure this will still mainly be a driver as an economic influence (i.e. if clouds work out cheaper then that will be the main driver) but it's nice to know that the cloud effect *could* be good for the planet too...

Outside of talking... my own cloud experiments are now mainly Microsoft Azure based. I've been running on a new Vista install and I've got some experimental code up and running. Also, using some recent Google read-ups I've been thinking about some of the ways to get around some of my problems with lack of "count" type operations in the non-SQL cloud - and I really do think offline incremental calculation may be the way... more "soon".

Ongoing Cloud Investigations

September was a busy month.

That may sound like an excuse... and I guess it is. However, my last few weeks of work were very busy, plus I managed to get away at the end of the month - managed to spend some time unwinding on holiday in Cuba.

But I've not forgotten this blog entirely - and now I'm back to give a brief update.


Firstly I've carried on reading about and playing with Google Apps Engine. The limits in the data storage continue to vex me. I kind of like statistics... and most of the web apps I want to write seem to involve data manipulation - including the use of GROUP BY type SQL clauses - and the use of COUNT, MAX, MIN, SUM. Because of the semi-structured nature of Google's BigTable (which are based on very good principles designed to help applications be extremely quick and scalable), these functions simply aren't available. The workarounds I've looked into for these problems are:

• Calculating all the required COUNT, MAX, MIN, SUM type stats in Python code when required - but these calculations are certainly not ideal - and not really scalable as a solution.
• Storing all the required stats in special BigTable entries - and updating these stats whenever a web request arrives to add new data, or to update or delete existing data. This is more efficient that the first solution - but requires a lot of additional code to be written and I'm worried it might be a particularly error prone set of code...
• Storing all the required stats in special BigTable entries - and updating these stats offline using some batch process - but this is quite awkard as the data store API is not particularly friendly for batch processes (e.g. the upper 1000 limit in the number of rows returned although this can be worked around) and because the Apps Enginer does not support batch processes directly - so you have to mimic them yourself using special web hits (but noting that each web hit can only consume at most 3 seconds of processing and that Google might block these requests for reasons of load balancing...)

In short, I'm more and more frequently considering reverting back to non-cloud application models for many of my planned apps.

Secondly, in the last week (while I was away), Microsoft has finally and fully unveiled it's Azure platform - it's answer to Cloud computing - to Google Apps Engine and to Amazon S3 and EC2 services. I've downloaded the Community Technolody Preview of this and got it partially working - good old Microsoft - they're only supporting Vista as a development platform initially! To speed up my curve I've also spent some time watching some of the PDC speeches - especially:

• Developing and deploying your first service - http://channel9.msdn.com/pdc2008/ES01/
  
• Best practices - http://channel9.msdn.com/pdc2008/ES03/

I'm also planning to watch:

• The keynote at some point soon - given by Don Box who's always good for talking, coding and *most importantly* not using Powerpoint -
• Some storage tech talks - http://channel9.msdn.com/pdc2008/ES04/ and http://channel9.msdn.com/pdc2008/ES07/

From what I've seen so far... microsoft have really hit the ground running hard here:

• Their solution looks very polished
• Their feature set is already more extensive in some places than Google's - e.g. they have provided a queue feature and further have provided facilities for background "worker" processing too.
• There are a few places, however, where there support is slightly less than finished - e.g. there were questions in some of the tech sessions about lack of secondary data indexing and enterprise developers were already pushing hard for more advanced transaction support and for more private features (how suitable is the cloud for private or enterprise data?)
  
• Their tool support is superb - there are some gaps where you have to drop down to command line tools, but already the level of Visual Studio integration is very significant - plus the team are clearly looking to push the newer MVC ASP application pattern into use for the cloud apps.
  
• Behind the generic Azure web application hosting service, there are further Microsoft services in production - including the Live services (for user login), and the SQL Data Services which would provide massively scalable (but not massively gigantically BigTable scalable) full SQL data services - which would specifically help with my COUNT, MIN, MAX, AVG stats.
  
• The marketing team really are already pushing hard - Microsoft is definitely 100% behind the cloud.

So what does this mean about my direction? Well, actually, having looked at the Microsoft products I am kind of tempted especially as:

• Having spent so long working on Windows and with Microsoft code and tools, then Visual Studio just feel like "home" and the Intellisense is so much better than Python's in Ellipse.
• The fact that Microsoft is so new gives me a chance to establish more of an early uniqueness - perhaps a chance to make a bit of a splash/name for myself (although my lack of .NET 3.5 experience, of LINQ. of WCF, of ASP.NET MVC, etc will all leave me quite some way behind at the start.
• I think there'll be lots of opportunities in both the .NET Cloud and in the Google Cloud (and beyond - Amazon are still a pioneer here, plus I haven't mentioned Rackspace and Mosso yet!) so it doesn't harm to try to use them
  

However, at a purely functional level I'm not convinced that I'll have any less problems with achieveing functional websites with Microsoft's Cloud solution instead of Google's - I think both are fairly similar and will show similar problems.

And, of course, I'm also very disappointed that the development tools for the Microsoft Cloud aren't yet Windows XP compatible...