Project Oxford – Image Text Detection

Microsoft has a new set of services which use machine learning to extrapolate data from images and speech called Project Oxford. Each service has a public facing REST api allowing any program capable of communicating through HTTP to utilize it, and for smaller amounts of data, Project Oxford is free to use.

Microsoft classifies these services into three categories: vision, speech, and language. The vision section contains several different capabilities including facial recognition, facial emotion detection, video stabilization, and optical character recognition (OCR). It’s OCR api allows a system to send an image URL and in turn will return the text it detected. Since the service is system agnostic, it’s possible to create an HTML page (with a little help from a service like Imgur) which can upload an image and extract the text without using server side resources. (The completed example can be found here.)

How to create

Uploading the Image

The first step is uploading the image to a publicity accessible store. Imgur works well for this, because it’s easy to use and free. It requires an account and registering the “application”, but this is relatively quick and painless. The page to do so is found here, and with the api key, all that is left is to retrieve the image from the input and upload the file.

Uploading the File

HTML 5 has the input type: file. To upload the file when the user selects it, add an onchange event to the input.

Now the function uploadImage fires whenever the user select a new file.

With the help of the FileReader, the webpage can load the image into memory to send to the server. It has several methods for retrieving files such as readAsText and readAsBinaryString, but the one which allows the image to upload correctly is readAsDataURL as this uploads the image in the correct encoded format.

Reading the file is an asynchronous operation and requires a method to call when loading. There is a property in the FileReader object named onload and has event parameter, but it’s not necessary in this context, because the reader’s result object is available through closure, so omitting it is fine.

Sending the File to Imgur

Whether using JQuery, another framework, or plain JavaScript, the upload process is straight forward. This example is in JQuery.

The Imgur api exposes the https://api.imgur.com/3/image URL, and to upload an image, use POST and add the HTTP header Authorization with the client id: Client-ID application_client_id. The trickiest part concerns the file upload from the FileReader. The readAsDataURL returns more data than necessary and the contents look like

The comma and everything before it is extraneous and will cause an error, so it’s necessary to remove it. Using String.Split(‘,’) and picking the second item in the array is the easiest approach.

Posting the image to Imgur returns several pieces of data about the object, height, width, size, etc., but the key pieces are, link (which is the URL to the image), and deletehash (this will be used to remove the image once completed. The api documentation says in some cases the id can be used, but since this is an anonymous posting, the delete hash is necessary).

Data OCR

Project Oxford requires a login to grant the necessary api key. After signing up and logging in, the service portal lists a page with access to the various API keys (there is one for each service). The once necessary for OCR looks like:

get api key for ocr

To retrieve the image text information, POST to: https://api.projectoxford.ai/vision/v1/ocr. It requires the HTTP Header Ocp-Apim-Subscription-Key with the value being the client id retrieved from the key dashboard. The data contains the URL as the key, and there are two optional querystring parameters which the request can have: language and detectOrientation.

language

The parameter language specifies which language to use when performing the OCR. If it’s not provided, the service attempts a determination based on the image and return what it thinks is correct. This can be helpful, if the language is unknown, and goal of the OCR process is to extract the text and then translate it. (Unfortunately, Microsoft has disabled the features, mainly Cross Origin Request Sharing, for retrieving the authentication token necessary to create an application not having any server side processing in it’s Microsoft Translator API). Not including leaves the service up to guess and can provide incorrect results.

The parameter is the BCP-47 language code which follows ISO 639-1. (A lowercase two character code representing each language e.g. en for English, es for Spanish, etc.) At this time the API only supports about 20 languages, but Microsoft has said it is working to expand the list. The API documentation lists the supported languages under the language parameter.

detectOrientation

This is a boolean parameter indicating it should attempt to recognize the text and orient it to be parallel with the top of image’s bounding box. Sometimes this can help in how it groups the returned text.

Response

The service returns an object looking like this:

language

The language property is what the service thinks the language in the image is, and if there are multiple languages in the text, it still only returns one. The service makes its best determination and won’t necessarily pick the first it encounters.

TextAngle

TextAngle is the tilt the service thinks the image is set to, and orientation is the direction the top text is facing. If the text is facing 45 degrees to the right, the text angle would be “-45.000” and the orientation would be “Right”.

Regions

multilanguage The found text in the image is not lumped together when returned. There are separate entries for each section of text found. In the example to the right, Hola and Hello are found in two different areas, so there are two regions returned in the array. Each region has a property boundingBox which is a list of comma separated coordinates where the text region exists on the page. In each region there is a lines property which is an array of objects each with their own boundingBox and each line has a words property object array separating each word and also containing the boundingBox property.

Removing the Image

Removing the image is similar to the initial POST action. This time the action is DELETE and the URL has the deletehash appended to the end.

The Javascript code can be found here.

Minutes and Seconds

When dealing with dates in .NET, most applications use the System.DateTime struct to store and manipulate dates. It has a natural conversion from MSSQL Server’s DateTime datatype, and has little difficulty in translating into different data formats (JSON, XML, etc.)

For a while now JSON has been a standard in communicating between applications, web clients and servers, to and from database and so on, and serializing a DateTime object looks like this converting it into the ISO 8601 format:

and converting it back works exactly as expected:

Serializing the date and parsing it with JavaScript on some browsers (such as Chrome for Mac and Windows, and Safari) yields different results. The browser has moved the date 4 hours into the past, because JavaScript has assumed the time it parsed was in UTC.

Secondly, JavaScript based its interpretation off of an incorrect timezone assumption. In 1923, the majority of the United States stayed on standard time the entire year, and those that moved their clocks forward did so on April 29. Daylight Savings Time was not in effect, and throughout history, the use of it, and when it was applied, has varied. Even if the passed in date was meant to be in UTC, the conversion should have made it Mon March 19, 1923 19:00:00 GMT -0500 (EST).

This also means when serializing the date to JSON and sending it back to the server, it sends it as it’s assumed UTC date.

At first glance, this looks correct, but changing the date to 1923-05-20T00:00:00.000Z (adding the Z at the end, indicating the date is in UTC) yields 3/19/1923 7:00:00 PM showing that the .NET DateTime object is trying to apply the Time Zone Offset. Now the question is, “Why is .NET deserializing the date as 3/19/1923 7:00:00 PM and not 3/19/1923 8:00:00 PM like JavaScript does especially since the serialized date came from JavaScript after it interpreted the original date as UTC?” The answer is because JavaScript only interpreted the date as UTC and displayed it as the timezone the machine is set in. The date it had didn’t change, so when it serialized it to the ISO 8601 format, JavaScript only appended the Z thinking the date was already in UTC.

Sending the Time Zone Offset when serializing from .NET, fixes the inconsistencies in some instances, but not all of them.

Since the time zone wasn’t specified, .NET assumes it is the local time of the server and produces 2016-03-11T19:00:00-05:00. Loading that string into a JavaScript Date object produces the same results although in UTC:

One of the issues with creating DateTimes in .NET and not explicitly setting the timezone is that it can make incorrect assumptions. Setting the timezone to Indiana (East) and parsing the following date 1989-05-12T00:00:00.000Z yields 1989-05-11T20:00:00-04:00 showing the UTC conversion to Eastern Daylight Time. Indiana didn’t follow Daylight Savings Time until 2006, meaning the conversion is incorrect. (Even if it did do the conversion correctly, setting the Set Timezone Automatically flag in Windows 10 forces it to be Eastern Time)

Explicitly setting the timezone using the DateTimeOffset corrects the problem in .NET, but it doesn’t guarantee correctness across systems.

When the browser parses the date with the correct offset, it attempts to display the date with the offset it thinks it should be.

The data in the Date object is still correct, and exporting it will not alter it in any way. Its attempt to display the date in a local time is the issue. In the United States, the only dates that are affected are prior to 2007, because that is when the last time a change to the timezones occurred. (The start of Daylight Savings Time was moved ahead to the second Sunday in March, and then end extended to the first Sunday in November. If you load a date from the year 2004 into JavaScript in Chrome, the date will incorrectly show for all dates between March 14 and April 4.)

The real solution to handling dates is to use libraries specifically designed to handle timezones and Daylight Savings Time (MomentJs for JavaScript and NodaTime for .NET). These significantly reduce the inconsistencies found in using the built in date and time conversions found in .NET and in Browsers, but even these can have mistakes in them.

Configuring Logic

This question talks about removing a switch statement so that every time the business logic changes concerning a multiplier value, the C# code itself doesn’t have to be changed and the application recompiled. I proposed loading the keys and multiplier values from a configuration file into a dictionary and accessing the data when needed. (The following example shows it loaded in the constructor for brevity.)

A comment in the answer mentioned the benefits of creating extra classes, and how the dictionary approach could not handle more advanced calculations should the need arise. With a slight modification, and some additional code, this no longer becomes a hinderance. Expression Trees allow the program to dynamically create functions and execute them as it would with compiled code.

Based on the question and the example above, the current equation has two parts, the travelModifier (which is determined by the mode of transportation) and the DistanceToDestination. These are multiplied together, and return a decimal. Completely abstracting this out into its own function (which then becomes the model to base the configurable functions from), would make the method look like:

Since the travel modifier already comes from the configuration file, it is unnecessary to pass that into the function, because when the application reads the configuration and creates the method, each entry will have the travelModifier value already coded into the function so that parameter can be removed, and an example function in C# would look like:

To accomplish this, each entry in the configuration file would need to have two parts, the method of travel (bicycle, bus, car, etc.), and the equation. The latter is a combination of the travelModifier constant, the distanceToDestination and operators (+,-,/,*). An entry in the file would look like this:

Before loading the configuration file, the dictionary which will hold the function and retrieve it based on the selected method of travel will need to be changed. Currently it has a string as the key and a double as the value:

Instead, it needs a function as the value.

Loading the contents from the configuration file has a few different steps. Retrieving and separating the parts, parsing the equation, and creating the method at runtime.

Loading the Configuration File and Separating the Parts

Parsing the Equation

It would be possible to parse the equation and immediately convert it to an Expression, but it’s normally easier to load it into an intermediate structure so data can be transformed and grouped into a usable structure first. The equation has three parts, and an enum can help distinguish between them.

and the class to hold the equation parts

In order to parse the equation, the program needs to determine what is an operator and what is a variable or constant and its execution order.

A Note About Math things

Execution Order

Consider the following: 2 + 4 / 2. At first glance, it looks like the answer is three, but that is incorrect. The multiplication and division have a higher operator order precedence and their calculations occur before addition and subtraction. This makes the actual answer 4. The C# compiler knows about order of operations and which happens first. When building the expression tree, the runtime doesn’t take this into account, and will execute each operation strictly from left to right. It is important to note this when creating and grouping the intermediate objects to form a tree with the execution order, so it is correct.

Making the Expression

The System.LINQ.Expressions.Expression is the class used to create the lambda expressions. The actual method to create the function is Expression.Lambda<T> and then call its compile function to turn it into a callable method.

The Lambda function requires two parameters, an Expression, and a ParameterExpression[]. The entries in the ParameterExpression[] are the parameters to the function and they are made by calling Expression.Parameter.

Expression Body

Each Expression object is a tree of Expression objects. The four methods used to create the operator functions (Expression.Add, Expression.Subtract, Expression.Multiply, and Expression.Divide) all take two Expression parameters (the left term and the right term), and each Expression can be one of three things, a constant (Expression.Constant), the supplied parameter (ParameterExpression), or another Expression.

With this, all that is necessary is to convert the EquationPart tree into an expression.

Additional Actions

It might be necessary to do additional actions in the expression, for example method’s output could be logged to the console. To do this, the Lambda Expression would now need to:

1. Calculate the result of the equation (calling the created equation).
2. Assign that value to a variable.
3. Write the variable contents out to the console.
4. Return the result stored in the variable.

Right now, the body of the Lambda Expression is the result of a single Expression object. All the actions culminate to a single result, but when adding logging, this changes. Calculating the result and logging it are separate unrelated actions. The Expression.Block groups Expressions together, and returns the value from the last executed Expression.

The first step is creating a variable using Expression.Variable it takes a Type and optionally a variable name.

Then assign the results of the body Expression to it:

Now the system can log the result, by using Expression.Call.

The Expression.Block method takes Expressions to be executed in the entered order. The only exception to this is the creation of the variable which much be passed into the method by a ParameterExpression[].

The full method with the console output looks like this:

If/Then

The methods use the double type resulting in the impossibility of a DivideByZeroException. Per the C# specification, it returns the value infinity.

To create a conditional statement use the Expression.Condition method which has three parameters (the Expression for the test, the true block, and the false block).

Test Condition

The test condition is an Expression, and the double type has a static method for checking for the infinity value. To use it, the Expression.Call method works just like it did with writing data to the Console.WriteLine.

True Block

If the condition is true (meaning that the value is infinity, then it should throw an exception indicating a problem. Expression has a method for throwing exceptions, Expression.Throw

Empty False Statement

A false statement isn’t necessary, because if the condition is false, it will continue to the next statement outside of the condition. The Expression.Condition will not allow null as the third parameter, so to have an empty false statement use Expression.Empty instead.

Try Catch

Instead of passing the exception to the calling method, a second option would be to log it first by wrapping the method contents in a try-catch block. The Expression.TryCatch method has two parameters: the expression which contains the body information in the try statement, and the CatchBlock. Expression.MakeCatchBlock has three parameters: the type of Exception the catch block is for, the ParameterExpression which allows the Expression to bind the Exception to a variable for use, and the Expression code inside the catch statement.

Expression.Rethrow

Expression.Rethrow has two method signatures. The first has not parameters, and the second has a parameter of type of Type. In this example, since it is the last statement in the catch block (the the statement in a block determines what is returned from the block), if you use Expression.Throw(), the application will return with this error: Body of catch must have the same type as body of try. This is saying that the the try and catch blocks must have the same return type. In the example, the try block returns type double, so the catch block must do the same. The overload for Expression.Throw(Type), tells the runtime “This catch statement will return this type if necessary.” Since it’s throwing the exception, it won’t ever return a value, but this tells the Expression generator this will be the intended behavior if an exception doesn’t occur.

Here are all the code examples.