Internal project – step 2 in progress

After a bit of a hiatus, I’ve returned to working on my internal project:

I’ve partially completed my initial planned step 2. I ended up spending a lot of time on setup activities, ie yak shaving. I’ve been trying to restrain myself, but it is hard.
Some of the yak shaving activities:
  1. Worrying about the structure of the project and which directory things belong in.
  2. Setup for a dev machine, in this case powershell scripts to create a database and some initial base data. I managed to avoid putting together deployment tools, but I have ended up with scripts to create the database.
  3. Implementing authentication by hand rather than just using something off the shelf. This was truly a bad idea, but by the time I’d realised it, I was already too far down the rabbit hole.
  4. Using a more secure hashing algorithm. While it was interesting to use BCrypt, it wasn’t really necessary.
However I did find something fairly interesting things during this process.

Future plans impact structure now

I was planning to swap out the backend for a nosql database. The aim was for this change to be seamless. However the structures that you might use in a relational database would be different to a nosql database.

So the Sql tables look like this:

  1. status (status_id, message, date, user_id…)
  2. status_view (status_id, date…)
  3. status_like (status_id, date…)
However for a nosql or document oriented database, it would make far more sense to have a single document represent each post, including all the data associated with the post.
This has a direct impact on how the code needs to be structured to support this. Rather than having a repository for each table, it needs to have a single repository interface to cover all of the interactions with status, ie something like this:

Structure

I took the time to split the code up between a number of projects in order to isolate the projects by functional area. I’ve currently got:
  1. DataInterfaces – models and interfaces for accessing any data
  2. DataSql – the sql implementation of the interfaces in DataInterfaces
  3. SiteLogic – the business logic for the application, all code here should be completely unit testable
  4. SiteLogic.NUnit – unit tests for the library
  5. Site – the MVC site
  6. Site.NUnit – the unit tests for the MVC site.

Testing

As far as possible I’ve been wanting to write unit tests for all of the code. Obviously this isn’t possible all the time, for example I gave up on unit testing the controller for authentication. 

Currently Reading – 2013-01-08

Updated reading list:
64 Things was rather interesting. It’s a rather non-technical summary of things we can expect in the future with technology and the internet. It’s a bit over a year old now, so things have shifted a little (bitcoin and the recent NSA revelations spring to mind). It is a pretty comprehensive summary and very readable. A good book to give someone who isn’t hugely technical.

Internal project – Architecture

I’m rather fond of Military history and one of the quotes that I like is:
Plans are nothing. Planning is everything. Dwight D Eisenhower

I realised I missed a step in my original plan for the project: planning. The reason I like this quote is that it is clear that the plan might not be followed, however the process of planning forces you to think about what you want to do.

So I’ve done some initial planning around this. Since I’m reading a book on documenting software architecture, I’ve dug into that for the most appropriate view to use.

Layered Architecture

Some notes on this:

  1. I want to clearly separate the UI layer from the business layer in order to facilitate adding an API
  2. The data access layer might be swapped out with a completely different data backend in the future, so that needs to be separated out. I’m currently thinking that I’ll implement these Entity Framework for the first version. I’ll avoid generic repositories to create a clean interface that doesn’t require implementing a LINQ interface over a NoSql database that might not cleanly support it.
  3. I want the Authentication to be clearly separate from the business layer in order to keep this as dependancy free as possible. This tends to be a rather complex area that is well worth ensuring it doesn’t have any dependencies.
  4. Cross cutting concerns like logging aren’t covered in this diagram

Currently reading – 2013-12-29

I find I tend to have a lot of books I’m planning to read but haven’t read. In order to keep myself accountable, I’ve decided to start posting the books I’m currently reading.

Internal Project – Step 1 – Document the UI

I’ve started putting together the UI for my home project.

You can see this in all it’s glory at https://github.com/dcamdupe/status.

Here is a sneak peek:

Home page version 1
Home page version 2
These high quality wireframes help illustrate a number of things:
  1. I don’t have access to appropriate tools at home. Rather than buying them, I’m using the highly technical approach of pen + paper + iPhone camera to capture my wireframes
  2. My hand writing is frighteningly messy
  3. In version 2 I’ve avoided the traditional wooden table for a far more attractive red table cloth with stars
For version 2 of the wireframes, I’ve tightened things up a little and used more consistent UI elements. This might not be apparent, given my limited ability with a pen. I also noticed I missed a page and a few other major elements on the pages.
I’m skipping a few normal user lifecycle elements (forgotten password etc) for simplicity’s sake.
The only thing that might not make immediate sense is the popularity box. This will be some sort of synthesis of the number of views and likes to provide an overall popularity. This could be represented by a number, an image or just a colour.

2014-01-05 Updated to link to earlier article.

Internal Project

I’ve decided to start a home project to test a few different technologies. The aim is to pick a a relatively simple project and implement this in a few different ways.

The simple project is a website where someone can post a status.

The rough outline is a website where:

  • Authenticated Users can:
    • post a status
  • Anonymous & Authenticated users can:
    • View current Status
    • View past statuses and search through them
    • ‘Like’ a Status

I guess you could call this twitter without most of the useful features.

I’m aiming to try a number of different technologies, in decreasing level of comfort.

I’ll also be swapping pieces in and out with new implementations in different technologies.

The current plan

  1. Document the UI – I’ve found this tends to make the implementation clearer
  2. Implement in bootstrap, MVC4 with SQL Server backend
  3. Swap the SQL Server backend for a No SQL database
  4. Design JSON API to access app
  5. Implement JSON api using WebApi backend
  6. Replace MVC app with javascript client side framework (currently planning to try angular)
  7. Replace the WebApi backend with an F# implementation
  8. Replace the WebApi backend with node.js
I want to apply best practices throughout, which might mean learning new best practices as I start using new technologies.

I’ll post all parts of the implementation and any documentation on Github.

Working with External APIs

I’ve done a fair bit of work with external APIs. It’s generally a fairly frustrating process. So I’ve collected a list of best practises. Some of these are shamelessly stolen from colleagues.

Some of these are a bit ‘heavy’ and not appropriate for all circumstances. This isn’t a checklist to follow blindly, it’s just a list of some good ideas I’ve picked up.

1. Logging

Log all calls going out and all incoming responses along with dates & times. Any logging should occur as close to the boundary as possible. This is to ensure that you remove as many layers of abstraction. For example if making webservice call, you would want to log the raw request & response, not a serialised version of the data you send.
Logging will help answer questions like:
  1. Response times, current and historical
  2. Is the API still available & responding?
  3. Has something changed at the remote end?
  4. Have we correctly translated internal data to external calls?

2. Mock the API

When trying to develop or test something that depends on an external API, you’ve got a huge dependency on that API. Development and testing can stall if that external API is unavailable. This is also an issue with intermittent faults. A lot of time can be wasted on trying to identify that that API is unavailable or is not working.
If the API has a test environment to work against, typically this is less stable than the production environment. Updates are shipped more frequently, issues are addressed more slowly than the production environment. Response times are also typically slower, as the environment is rarely provisioned like production.
Mocking the API protected you from all of these. Typically this means capturing known good responses. When the mock API is enabled return, these known good responses. This can be extended further to trigger specific error conditions when certain data is passed in, eg a specific customer name will trigger a duplicate name error.
You will need a simple way to switch the mock API on/off, eg a simple config file change.

3. Write Functional tests of the API

This might seem a bit overboard, but doing this can save a huge amount of time. Typically we end up doing this sort of testing manually, but it is worth doing this more systematically.
Writing functional tests helps explore the limits of the API. It can help expose bugs in the API and clarify things that might be unclear from any documentation. It can help to benchmark performance. It can also be a great source of data for mocking the API.
It can also be a great source of protection from changes in that API. If something fails, you can simply run the functional tests to verify whether anything has changed.