Open-source FTP-to-Azure blob storage: multiple users, one blob storage account
A little while ago, I came across an excellent article by Maarten Balliauw in which he described a project he was working on to support FTP directly to Azure’s blob storage. I discovered it while doing some research on a similar concept I was working on. At the time of writing this post though, Maarten wasn’t sharing his source code and even if he did decide to at some point soon, his project appears to focus on permitting access to the entire blob storage account. This wasn’t really what I was looking for but it was very similar…
My goal: FTP to Azure blobs, many users: one blob storage account with ‘home directories’
I wanted a solution to enable multiple users to access the same storage account, but to have their own unique portion of it – thereby mimicking an actual FTP server. A bit like giving authenticated user’s their own ‘home folder’ on your Azure Blob storage account.
This would ultimately give your Azure application the ability to accept incoming FTP connections and store files directly into blob storage via any popular FTP client – mimicking a file and folder structure and permitting access only to regions of the blob storage account you determine. There are many potential uses for this kind of implementation, especially when you consider that blob storage can feed into the Microsoft CDN…
Features
- Deploy within a worker-role
- Support for most common FTP commands
- Custom authentication API: because you determine the authentication and authorisation APIs, you control who has access to what, quickly and easily
- Written in C#
How it works
In my implementation, I wanted the ability to literally ‘fake’ a proper FTP server to any popular FTP client: the server component to be running on Windows Azure. I wanted to have some external web service do my authentication (you could host yours on Windows Azure, too) and then only allow each user access to their own tiny portion of my Azure Blob Storage account.
It turns out, Azure’s containers did exactly what I wanted, more or less. All I had to do was to come up with a way of authenticating clients via FTP and returning which container they have access to (the easy bit), and write an FTP to Azure ‘bridge’ (adapting and extending a project by Mohammed Habeeb to run in Azure as a worker role).
Here’s how my first implementation works:
A quick note on authentication
When an FTP client authenticates, I grab the username and password sent by the client, pass that into my web service for authentication, and if successful, I return a container name specific to that customer. In this way, the remote user can only work with blobs within that container. In essence, it is their own ‘home directory’ on my master Azure Blob Storage account.
The FTP server code will deny authentication for any user who does not have a container name associated with them, so just return null to the login procedure if you’re not going to give them access (I’m assuming you don’t want to return a different error code for ‘bad password’ vs. ‘bad username’ – which is a good thing).
Your authentication API could easily be adapted to permit access to the same container by multiple users, too.
Simulating a regular file system from blob storage
Azure Blob Storage doesn’t work like a traditional disk-based system in that it doesn’t actually have a hierarchical directory structure – but the FTP service simulates one so that FTP clients can work in the traditional way. Mohammed’s initial C# FTP server code was superb: he wrote it so that the file system could be replaced back in 2007 – to my knowledge, before Azure existed, but it’s like he meant for it to be used this way (that is to say, it was so painless to adapt it one could be forgiven for thinking this. Mohammed, thanks!).
Now I have my FTP server, modified and adapted to work for Azure, there are many ways in which this project can be expanded…
Over to you (and the rest of the open source community)
It’s my first open source project and I actively encourage you to help me improve it. When I started out, most of this was ‘proof of concept’ for a similar idea I was working on. As I look back over the past few weekends of work, there are many things I’d change but I figured there’s enough here to make a start.
If you decide to use it “as is” (something I don’t advise at this stage), do remember that it’s not going to be perfect and you’ll need to do a little leg work – it’s a work in progress and it wasn’t written (at least initially) to be an open-source project. Drop me a note to let me know how you’re using it though, it’s always fun to see where these things end up once you’ve released them into the wild.
Where to get it
Head on over to the FTP to Azure Blob Storage Bridge project on CodePlex.
It’s free for you to use however you want. It carries all the usual caveats and warnings as other ‘free open-source’ software: use it at your own risk.
If you do use it and it works well for you, drop me an email and it’ll make me happy. 
OCZ Colossus Performance in AHCI vs IDE mode on the Intel DP55KG Desktop Board
For those of you who don’t know, I’m currently building a new high specification PC to handle the day-to-day rigours of software development. Like a lot of developers, I’ve decided to embrace the new SSD drive technology due to the massive performance increase to be had over traditional ‘plate drives’.
I’ll provide more details about the PC’s specification soon, but I wanted to share some information about the various ways in which SSDs can be configured and how they affect performance. There are plenty of articles out there regarding SSD design, from a technical point of view, but I haven’t found any that have compared AHCI to IDE so that I could figure out which might give my Colossus the best chance of performing well. So, I decided to run my own basic test.
Motherboard
For this test, I am using the Intel DP55KG motherboard running BIOS version KGIBX10J.86A (17th Feb 2010). The DP55KG has 8 on-board SATA ports, six of which are powered via the board’s PCH, and two via an integrated Marvell 88E6145 chip (check out this post for an excellent review of the DP55KG).
The SSD
The drive under scrutiny here will be the OCZ Colossus 120GB. On paper, according to OCZ, the drive supports read and write speeds up to 260MB/sec, with sustained write around 140MB/s. This was the best drive I could find of this capacity in the price bracket.
The 3.5" 'Colossus' SSD by OCZ
The test
I wanted to know, “Which performs better, AHCI or IDE?” followed by “will IDE mode on the Marvell chip out perform IDE mode on the Intel PCH controller?”. To find out, I will use Passmark Performancetest 7.0 from Passmark. It provides a simple, standardised means of testing my SSD’s.
- To test the Intel PCH:
- Configure the controller in IDE mode
- Install Windows 7 Ultimate
- Install the latest Intel Chipset drivers
- Restart
- Install Passmark PerformanceTest 7.0
- Run tests
- Repeat these steps, but next time configure the controller in AHCI mode.
- To test the Marvell Chip:
- Connect the OCZ Colossus to port 0 on the board
- Verify controller is in IDE mode (AHCI is not supported by the Marvell chip)
- Install Windows 7 Ultimate
- Install the latest Intel Chipset drivers
- Restart
- Install Passmark PerformanceTest 7.0
- Run tests
Results
Here are the raw test results. Note that the figures quoted are in MB/s.:
| Test Name | Intel PCH (AHCI) | Marvell (IDE) | Intel PCH (IDE) |
| Disk – Sequential Read | 109.9 | 94.3 | 111.3 |
| Disk – Sequential Write | 131.0 | 46.1 | 131.8 |
| Disk – Random Seek + RW | 41.8 | 37.2 | 41.4 |
| Disk Mark | 1022.7 | 642.1 | 1029.0 |
| PassMark Rating | 2372.6 | 1489.7 | 2387.3 |
Here’s the corresponding bar chart (click to enlarge):
Intel DP55KG PCH IDE vs AHCI, vs Marvell IDE
How to make your Xbox 360, Windows Home Server, Windows Media Centre and Windows 7 work together seamlessly
In this post, I’ll discuss my experiences on building a robust, high-speed and high-definition media setup in your home, using Windows 7, Xbox 360 and Windows Home Server. If you’d like to leave a comment when you’ve read this post letting me know what you’d like to see more of (or less of), I’ll make sure I address that in either an edit to this post, or a brand new one. Thanks!
In this article, I cover:
- Introducing Windows Media Centre
- How To: Set up Windows Media Centre
- How To: Connect your Xbox 360 to Windows Media Centre
- Windows Home Server and your digital media
Feel free to skip to the section that interests you most.
An impulse purchase that changed the way we consume media in our household
At the weekend, something happened that rocked my world. It didn’t happen accidentally, either. I was out 
shopping and I just happened to walk past Maplin (RadioShack, to you US folks) who had a special offer in their window: a KWorld PlusTV DVB-T Dual Freeview HD Tuner for the princely sum of just £24.99. That’s a small USB stick with an antenna connection on it that contains not one, but TWO HD Freeview TV tuners, giving the ability to watch one programme while recording another: for £24.99.
In the UK, Freeview gives access to about 100 digital channels, and, although they don’t broadcast in HD yet, in early 2010 the HD rollout begins.
Right now, the only way to get free HD TV over the air is to get Freesat HD or Sky+ HD, both of which require a satellite dish to be installed and cost a fair amount in set up and subscription fees.
At the time, like many geeks, I didn’t know why I wanted this particular gadget, but I the little voice inside of me compelled me to buy it. I figured, “hey, even if I don’t know what to do with it, perhaps Windows 7 will”. And that is where the fun began.
Introducing Windows Media Centre
With the recent release of Windows 7, millions of people globally have been busy remarking how much simpler the ‘simple’ things become when you install Windows 7.
Unlike Apple (for whom I shall now depart momentarily to direct a slight dig) who only have to worry about a limited number of devices on a limited platform of hardware, Microsoft have to build software that is actually compatible with hundreds of thousands of different devices and at least twice as many combinations. That is by no means an easy thing to accomplish, but Windows 7 seems to do it with dare I say it a certain degree of sentience: plug and play has actually come to life. Everything… every single USB gadget I own was detected first time, without me first needing to supply a driver CD. If Windows didn’t have the drivers, it ‘magicked’ them off the internet for me. Marvellous.
This includes the £24.99 TV Tuner from some anonymous company I’ve never heard of before. I plugged it in, and up popped the “Driver Installation” window and it downloaded the relevant drivers for me (I’m so glad it did as well – I don’t like installing drivers and software from anonymous small name companies).
A few minutes later, everything was successfully installed with zero effort on my part. In fact, the most strenuous part of this entire exercise was trying to open the box the device came in – which was apparently designed to survive a small tactical strike.
And this is where Windows Media Centre comes onto the scene – an incredible suite of technologies that’s designed with simplicity in mind. Shipped with Windows 7 Home Premium and Ultimate editions, Media Centre has – at its core – the ability to act as a centralised hub for all your digital media: music, videos, recorded and live TV. For anyone who’s seen TiVO or Sky+, Windows Media Centre does exactly what those two services do, but better, faster and subscription free.
Setting-up Media Centre
Launching “Windows Media Centre” from your Start Menu brings up a little animated Window:
Having detected that I had now installed a USB TV Tuner, Windows Media Centre asked me if I wanted to configure it. Allow it to complete the setup process, which – including the time it takes Media Centre to scan the TV frequencies in your country for available channels – took me about 10 minutes.
When the process is completed, Windows Media Centre is able to give you a TV guide showing weeks’ worth of programming schedules. The guide alone is worth ditching Sky+ for:
- Beats Sky’s “Series Link” function by adding numerous extra functions giving greater control over what is recorded and when
- Search for TV programmes by keyword in the title, or by cast members
- Create wish lists: specify your favourite actors and actresses, and the guide records any programmes it finds with those stars as cast members
- Manage potential recording conflicts before they arise and set priorities for your favourite shows
The list does actually go on. Have a look at some of the guide screenshots below:
The trouble is right now – this is TV on my PC. My PC is in my office – not my front room. We need to break out of here and I’ll be damned if I’m going to run a VGA cable from here to my 42” LCD TV in the front room. Nay – I want an easier way!
Xbox 360
Those of us with an Xbox 360 (or three) laying about the house know that we’re supposed to be living in some futuristic media-utopia where our TV’s can access all our music, videos and photos. Even TV (alright, that’s slightly pointless but you get the picture). But, even with Vista, this wasn’t particularly easy to set up. So I never bothered with it. It just didn’t quite work as expected.
Turn on your Xbox 360 while your Windows 7 PC is on, and you’ll receive a polite little notification that “Windows has discovered a Media Centre Extender”. Discovery – what an appropriate word. It popped up, and I thought to myself …. “Please, continue”. And so Windows did. After being informed a few seconds later that everything was installed correctly, I wanted my Xbox 360 to be able to access the juicy goodness of Windows Media Centre.
Follow these steps:
- Go to “System”
- Go to “Computers”
- Go to “Windows Media Centre”
- If you’ve already set up (or tried to set up) a Windows Media Centre connection before, you’ll first want to choose “Disconnect”. If you’ve not tried that before, that option won’t be available to you.
- Now, choose “Connect”.
- Your Xbox 360 will search your network for available Windows Media Centre computers. After a few moments, your Windows 7 Media Centre should be discovered, and your Xbox 360 will display a two-part eight digit code on your screen.
- Note that code down, and fire up the Windows Media Centre application on your PC.
- Go to “Tasks”, then “Add Extender” to bring up this screen:
- Type in the eight digit code you were given by your Xbox 360, and press “Next”.
- You’ll then see a screen similar to this:
- This process might take a little while to complete while your Media Libraries are constructed.
- Once complete, you’ll see a screen like this:
- By this time, your Xbox 360’s display will have changed and will be displaying a very similar (if not completely identical) display to the Windows Media Centre application on your PC.
And that’s it, you’re done – your Xbox 360 is now connected to your Windows 7 Media Centre.
What about Windows Home Server?
Ah, the “server in the home”. Got one? I have. If you haven’t, you should.
It takes care of backing up all your PCs, laptops and personal devices and my HP MediaSmart Home Server even automatically backs up my iTunes Libraries and streams them over the network. In fact, they’re so good I’d recommend them even for small businesses. Just plug them in, install the software on your PCs and laptops and you’re safe from day one (you can even do a bare-metal rebuild if you are running Windows 7 – that rocks).
Even cooler is that with the PowerPack 3 Beta recently released by Microsoft you can now install a connector for Windows Media Centre that allows it to take full advantage of the awesome storage capacity of your Windows Home Server.
The Windows Home Server Media Centre Connector (desperately in need of a shorter name, though it is hard to imagine one more descriptive) adds a little tab to your Windows Media Centre menu that, in a nutshell, allows Media Centre to archive your recorded television programs off to your Home Server automatically. It also adds some other nifty functionality to allow you to view most of the information contained on your Home Server Console on your TV, too.
In summary
It starts with the purchase of a TV tuner ‘dongle’. With Windows 7, in literally just a few minutes you can pretty much overhaul the way you access your digital content in the home. Chuck an Xbox 360 into the mix, and you’ve got the capability to stream HD content to your TV, listen to your music and browse all your photos right from the comfort of your armchair. I’d also recommend getting your hands on a great media remote (this one from Logitech is awesome) because you’ll quickly get bored using the Xbox 360 controller. The Logitech remote can also control many other devices so you can ditch the collection of remotes you’ve probably already got.
With Windows Home Server chugging away too, you’ll never lose any of your digital recordings again and since storage is so cheap these days, you’ll have a very easy mechanism for expanding your storage on demand.
Microsoft unveil “Websitespark” programme
Microsoft recently announced the launch of the latest addition to their Partner Network: Websitespark. Following on from the success of the Bizspark programme for startup companies, Websitespark is aimed at supporting professional web development and design companies to succeed by offering proactive assistance in the form of business opportunities and it’s class-leading Windows and SQL Server products. So, whether you’re the “one man band” or a team of up to ten employees, you should check this out.
BizSpark has already helped hundreds (thousands, now, perhaps?) of startups to succeed, and Websitespark looks set to do the same for smaller teams of web professionals. With no up-front costs and no long-term commitments (just a $100 exit fee at the end of the programme period) , what more could you ask for?
Full information is online and available at http://www.microsoft.com/web/websitespark/
4 comments