Functional Languages in the Workplace

On a semi regular basis, people question why I choose to use F# to implement projects. They question why use a lesser known language when one like C# has a larger developer pool and is more widely documented. I explain to them my rational behind it, siting personal experience, and documented cases about others success stories as well. There is significant evidence showing functional languages can reduce commonly occurring defects due to their inherent nature of immutability provide easier support for scalability, and have a stronger type system allowing for more expressive code. There are numerous testimonials on the use of functional languages and their benefit, but after hearing all of this, they are still doubtful about even considering a change. Assuming this evidence is correct, the question of “Why isn’t this a serious choice for the majority of organizations?” continues to appear.
During discussions about switching to a functional language, I repeatedly hear the several common questions and arguments for resisting change. Most of these embody fear, uncertainty, and doubt. Several can be applied to moving to any technology, and although they should be considered, they are nothing which cannot be overcome. Here are my responses to the most common arguments against change I receive.

Our code is already written in language X, and it will be hard to make a change

There will always be legacy code, and it probably deviates from the standards used today. Was it written in a previous version of the currently used language? Does it contain libraries that are no longer supported? Was it written in such a way that converting it to current standards is difficult or impossible? If the answers these questions is yes, that doesn’t mean that other projects suffer the same fate.
Legacy code can’t hold you back from technological advancements, and it most likely doesn’t now. Over the last several years many software vendors have made sweeping changes to languages and technologies leaving them looking only vaguely like what they did when first created. The introduction of Generics, the inclusion of Lambda Expressions, and asynchronous additions made huge advancements in several different languages and greatly changed common approaches to solving problems. These enormous changes didn’t stop organizations from modernizing their many of their applications to take advantage of new features even though code written with them is radically different than in previously created applications.
Radical shifts in technology happen all the time, and almost every organization shifts its strategies based on trends in the industry. Organizations which defer changes to their current approach often find greater difficulty in migrating the longer they wait due to the fact that they continue to implement solutions using their current approach. Mindlessly shifting from one approach to another is never a wise decision. That introduces chaos, but neglecting trying new approaches due to legacy concerns can only end in repeating the same mistakes.

Our developers don’t know language Y. It will be too hard and costly for them to learn and migrate.

A developer’s job is to learn every day. There are new features to understand, new architecture patterns to master, and new languages to learn. The list is endless. The belief that at any stage in one’s career the road to deeper understanding ends, is myopic and ultimately an exit ramp to another profession or a stagnant career. Developer’s should be challenged. Organizations should push their staff to understand new things, and compared to the opportunity cost of repeating the same mistakes, the amount of time and money required to train people is often negligible, especially with tools like books, video learning, computer base training, etc.
There are some people that have no desire to continue learning, and that’s ok. New development isn’t for everyone, and going back to the previous point, there are always applications in need of support that won’t or can’t be converted. Organizational migration to a technology is almost never an all or nothing approach, and some solutions should be left exactly how they are, because of the cost of converting them will outweigh the benefits. There will be people to maintain those in the long term, and these solutions cannot be the bedrock against advancing how other projects progress.

What if we fail and we are stuck with a language we can’t use?

If an organization takes the leap of faith and switches to a functional language what is the probability of some failure during the process? The initial answer is, 100%. Everyone fails every day at something. Failure is inevitable. With this in mind, you’re already failing at something, so the question is what are you going to do to try and fix it? You’re going to create other problems too, but with planning, retrospective analysis, and learning from those mistakes, those will be solved as well, but ultimately the position you end at will be further along than where you started.
A few years ago, I had a discussion with an organization about their development practices. They were extremely adept at knowing where their time was allocated: support, feature enhancements, refactoring, etc. When asked about their breakdown, they explained on average 30% of their time went to fixing production defects from previous releases. They were perplexed about why they were missing deadlines despite becoming stringent on code quality. I asked about their plan to fix it, and they responded with a few ideas, but their final answer distilled to, “write better code.” When confronted with the question, “What are you going to change?” they said, “Nothing. Changing the development process is too time consuming and costly. If we update our practices, we’ll fall further behind on our releases.” The definition of insanity is doing the same thing and expecting a different result, yet several organizations believe they can break the cycle simply by standing still. If changing how an organization develops isn’t feasible, then changing what they develop with is one of the only few viable options remaining. It is much easier to change a technology than it is to change an ingrained culture, which is exactly why using languages and tools that enforce practices which reduce errors is a much more efficient approach than convincing everyone to work in a certain way.
Most organizations resistant to change perceive technology migrations as a revolutionary approach. They firmly believe all use of a certain technology immediately stops and the new one begins, because it is much easier to think in terms of black and white (one vs. the other) when change is a rare and uncomfortable occurrence. Change to anything new should be a cautious approach and take practice. It should be evolutionary. Organizations should try several smaller variations of an approach, learning from each and refining their ideas on gradually larger projects. Embracing a adaptation and “failure leads to a stronger recovery” approach ultimately leads to a better outcome.
It is almost certain moving from to a functional language from an unrelated paradigm is going to be difficult and confusing, but the fault does not lay to the language itself. As with anything new, the concepts are unfamiliar to those starting to use it. There will be mistakes during the learning process, and some projects will probably take longer than expected, but basing the long-term benefits on the first attempt to implement anything will show biased result against it, and with time moving to an approach which aids developers to make fewer mistakes and write better and cleaner code will save both time and money.

It’s not widely used enough for us to find people to support it

My coworker recently attended two meetups concerning functional programming, each having approximately 25 attendees. After the first one, he decided to do an experiment at the second. He asked people at the meetup, “How many of you use a functional language at work?” and the result was astounding. Only one person admitted to it, and it was only part time. At a minimum, there are 25 people at each location that are excited enough about functional programming to attend a meetup on their own time on a topic which has nothing to do with the tools they use at work, and these people are only a representation of the larger workforce. There are many others that were either unable to attend, or were unaware of the event.
There is almost no place in the United States that isn’t a competitive market for development staff. Large companies are able to pay higher rates and have better benefits which means they will pull the majority of the highest qualified candidates. Smaller organizations can’t offer the enormous benefits packages placing them in a difficult situation to fill needed positions. Picking a technology where there are fewer people to fill the role would seem to place those organizations at a disadvantage, but this is actuality in comparison to overall demand for those type of people. Looking solely at the number of potential applicants, the pool of functional programmers is smaller, but organizations using functional languages aren’t nearly as widespread, so they suffer less completion when searching for candidates. Furthermore, assuming the statistics surrounding the benefits of functional languages are correct, organizations will require fewer programmers accommodating the constraint of a smaller pool of applicants.

Conclusions

Functional languages can be an excellent fit for organizations, both ones starting development and others which have been established for a considerable length of time. Most resilience in using them comes from misunderstanding the benefits compared to the cost of changing languages. It is neither difficult nor time consuming to attempt to better the development process by focusing on tools to better aid the process.

C# 7 Additions – Deconstructors

C# has a new type of method, the Deconstructor. When a type implements this method type with the name of Deconstruct, multiple variables maybe directly assigned as a return type would.

The method must be named Deconstruct and have a return type of void. The parameters to be assigned all must be out parameters, and because they are out parameters with a return type of void, C# allows the Deconstruct method to be overloaded solely based on these parameters. This is how the new System.ValueTuple allows it’s properties to be assigned to separate variables without assigning each one individually.

Deconstruct also does not need to be directly attached to the class. C# allows the method to be implemented as an extension method as well.

At the moment it is uncertain if wildcards will be added allowing unneeded variables to be omitted from being assigned. This addition would allow the insertion of the * to indicate a parameter is not needed (similar to _ in F#)

C# 7 Additions – Local Functions

In C# 7 it is now possible to create a function within a function termed a Local Function. This is for instances where a second function is helpful, but it’s not really needed in the rest of the class. It’s created just like regular functions except in the middle of another function.

Just like normal functions, you can create expression bodied members as well

Local variables in the outer functions are accessible, and it’s possible to embed local functions inside other local functions:

So how does it work? Looking at the IL code, the compiler has converted the internal function into a private static one inside the class.

IL Code showing private static function

The name is generated at compile time, so it is not accessible to other methods, but it is still possible to access it through reflection with the private and static binding flags.

reflection shows local function.

Someone I know asked what would be a good use case of Local Functions vs. Lambdas. Lambdas can’t contain enumerators, and by encasing an enumerations in a local function it allows others parts of the outer method to be eagerly evaluated. For example, if you have a method which takes a parameter and returns an enumeration, the evaluation of the parameter won’t occur until program starts to enumerate the collection. Encapsulating the enumeration in a local function allows the other parts of the outer function to be eagerly evaluated. You can find an example of the difference between using one and not using one here.

C# 7 Additions – Tuples

In C# 7 Microsoft has introduced an updated Tuple type. It has a streamlined syntax compared to it’s predecessor making it fall it look more like F#. Instead of declaring it like previous versions, the new Tuple type looks like:

Likewise to declare it as a return type, the syntax is similar to declaring it:

The first thing to note about the new type is that it is not included automatically in a new project. If you immediately use it, you’ll see the following error.

As of VS 15 preview 4 (not to be confused with VS 2015), you must include the System.ValueTuple Nuget package to take advantage of it.

This raises the question about how the new Tuple type and the previous one included since .NET 4 are related? They’re not. They are treated as two different types and are not compatible with each other.  System.Tuple is a reference type and System. ValueTuple is a value type.

So what are advantages over the previous version? The syntax simpler, and there are several other advantages.

Named Properties

In the System.Tuple version, properties of the return object were referenced as Item1, Item2 etc. This gets confusing when there are multiples of the same type in the Tuple as you have to know what position had which value type.

Now it’s possible to explicitly name the item types to reduce confusion.

The Item properties (Item1, Item2, etc.) have also been included allowing methods to be updated to the new type without breaking code based on it’s predecessor.

It’s also possible to explicitly name the values when creating the object:

Deconstruction

It is now possible to name and assign variable values upon creating (or returning) a tuple. Although not necessary, it reduces the amount of code necessary to pull values out of the type.

It’s not certain if C# will get wildcards like F# to automatically discard values which aren’t needed. If they are allowed then it’s possible to only create a variable for the name like so:

Updating Values

System.Tuple is immutable.  Once created it’s not possible to update any of the values.  This restriction has been removed in the new version.  From a purely functional perspective this could be considered a step backwards, but in C# many people find this approach more forgiving and beneficial.

Like all value types, when it is passed into a method, a copy of the tuple is created, so modifying it in the in the method does not affect the original.

However if you compare two different tuples and they have the same values, the Equals method compares the values in each and if they are all equal, it considers them equal.

Integrations with Other Languages

Unfortunately, C#’s new tuple type doesn’t automatically allow it to translate tuples from F#.

F# can’t desconstruct the values like it can with it’s native tuples, and to return it, you have to explicitly instantiate the object type and add the values.

Either way, the translation to F# isn’t horrible as it acts like any other object passed to it by C#.

Compile Time Stored Procedures

(code for this project can be found here)

One of the largest problems with interacting with databases is the lack of compile time assurance that the application code and the SQL code interact correctly. Testing helps with finding the errors, but as a system grows in size and complexity it becomes both difficult and costly to find every mistake. There are pre-made solutions such as F# Type Providers or Entity Framework, but these present their own challenges such as not having the project in F# or avoiding the heavy handedness of Entity Framework.

Fortunately SQL server provides enough information about Stored Procedures in the Metadata tables that makes it possible to create a from scratch compile time safe mapping between stored procedures and application code. Combining this with
T4 Text Templates it’s possible to automatically generate all the code necessary to handle stored procedure inputs and returns.

The first part is to pull all the stored procedure data from the sys.procedures view.

I remove all the procedures starting with sp_ as this prefix is meant to designate that it is a system procedure for use by SQL Server. The three important pieces from the table are the Procedure Name (p.name), Object Identifier (Object_Id), and the Schema Name (s.name).

Select Sys Procedures

Armed with the object id it's now possible to pull all the data for a procedures parameters from the sys.parameters view.

With this and the procedure's name it's possible to construct code to call SQL Server stored procedures which is compile time safe. Simply generate a method which takes in the appropriate parameters to call the stored procedure. The last step is to generate a lookup table which translates the SQL datatypes to ones which are valid C# syntax.

Idenfitfing which parameters have a default value and be not passed to the procedure

Since stored procedure parameters can have default values and therefore aren't necessary to populate, it would be nice to be able to know which one's are available but aren't required. This is a little trickier. There is a field has_default_value which looks like it would tell you if the parameter has a default value and therefore is optional, but there is a problem in the SQL Server metadata table, and it is only correct for CLR parameters, and not SQL ones. In order to actually find out if parameter has a default value, there are two options.

  1. Use Sql Management Objects (SMO)
  2. Pull the stored procedure code using something like SELECT OBJECT_DEFINITION (PROCEDURE_OBJECT_ID)

Using SMO to pull the definition is actually relatively simple assuming you have access to the appropriate dlls from Sql Server Management Studio.

Code:

If you don’t have access to the SMO dlls, you can still use something akin to a regular expression (or some other parsing method) to pull the parameter information out from the Object_Definition. As with parsing anything, there are a number of gotchas to watch out for, but it is possible with a little bit of work.

User Defined Tables

Starting in SQL Server 2008 it is possible to pass tables as parameters into stored procedures. There is view called sys.table_types which lists all of the user defined tables and has the necessary columns to link it back to which stored procedures use it as a parameter, and retrieve its column definition.

Stored Procedure Output

To automatically generate C# code for SQL Server output, you’ll really need SQL Server 2012 and later for it to be effective. In that edition they added, the dm_exec_describe_first_result_set_for_object procedure

This procedure takes a stored procedure Object_Id and returns the column information concerning the first result set it finds.

Result Set

The great thing about this procedure is that not only can you generate classes which can automatically update their types and names based on a result set that changes during development, but the procedure will return nothing if the stored procedure is non functional because of a changes in the database. By running this this and having it map all the result sets, the ones which come up blank which should return data indicate the procedure will not work with the current schema and needs to be fixed. It quickly helps eliminate the difficulties of knowing which procedures will break during program execution.

Putting It All Together

With these procedure, all that is really necessary is something that opens a connection to the database, pulls the necessary data and create the appropriate C# files.

As an example and a quick starter I’ve created an example solution. The SQL metadata retrieval is in the Classes.tt file, and all the code C# generation is in the StoredProcedures.tt file. The database I used to generate the example code is found here, and uses the Adventure Works Database.

Return From The Void

In a recent project, we were tasked with making a service layer more manageable and efficient, and with a tight deadline each minute needed to count. My teammate and I quickly moved methods around, created new classes where necessary, and implemented the Dapper object mapper. About half way into the final testing, my coworker looked at me and said, “The end date for a workflow isn’t updating. Do you think it’s a caching problem?” After looking through the code, I quickly found the issue. The SQL update procedure inside the update function wasn’t called. Fortunately, it was a simple fix, but it occurred to me we used this pattern dozens of times in the service tier. How many other places was this exact problem occurring that we haven’t found?

Testing caught the errors, but, the time spent tracking down this type of issue should have been avoided. I designed the update functions to return void instead of boolean or integer and relied on the implicit success of a method which solely performs a side effect (which functions that return void are) to indicate it executed correctly. I forgot when a method doesn’t throw an exception, it doesn’t mean the intended action was successful but only that it didn’t explicitly fail.

Take for example, the following two functions:

and

To the calling function, there is no difference. They both take the same parameter and return the same type (or not at all in this case), but to the overall system, there is a huge difference. One correctly updates the value, and the other doesn’t do anything and indicates nothing is wrong.

Applying the functional principal of always having an output from a function easily helps this issue:

and

Now the application won’t compile until the function is written to perform an action. The compiler helps enforce correct code.

In truth, there are ways to get around the compiler error and still have a non-functioning program. This is a common pattern seen over and over:

Comment out the change to numberOfRecordsUpdated, and the compiler states the code is valid. By applying another functional pattern, this problem code is also precluded. Enforcing the rule that variables should be immutable, removes this problem pattern since numberOfRecordsUpdated couldn’t be changed once it’s instantiated which means that it can’t have a default initialization of zero.

Cut the Header Off and the Body will Die (Well not really)

Recently, I had to work on an audit item concerning removing the “X-Powered-By” header from the IIS websites.
http header

You can right click on the header and select remove, but this causes two problems. One if you have a lot of sites (I had about 30), it becomes rather time consuming. Two you’ll notice that the Entry Type of it is set to Inherited. With HTTP headers there are two entry types in IIS: Local and Inherited. Local means the current site or web application is the point of origin. If you delete it, the header will be removed from the current node and all it’s children. Inherited means just the opposite. It is not the point of origin and removing it does not permanently delete it from IIS. All web sites list this response header as inherited.

You can click on the machine which hosts the sites in IIS and remove it:

Delete From Computer

The problem with this being that someone has to manually delete it from each server. This is both time consuming and error prone (any manual task is), and with PowerShell there is a much easier way:

Http Header Removed

Consequently, if you want to check to see if it exists before running the delete command, you can add a few lines to check. By default the Remove-WebConfigurationProperty will output a warning if you try to delete something which doesn’t exist.

For any IIS Administration to work, you have to run PowerShell in Administrator mode. With a hardened server you’ll have to supply your credentials to do so.

PowerShell has the ability to add headers as well. Just use the Add-WebConfigurationProperty.

JSIooa nvrtai Sncgtpi -> Sorting In JavaScript

JavaScript has built in functionality to sort arrays, but it doesn’t always work the way you think it would.

Take the following array:

Using the Array.Sort function (which all arrays have because of inheritance)

Orders the array in the following order:

Default Sort Running Example

Even though the array is numerical, the default Array.Sort places the 15 first in the series. The default comparison function sorts values by their Unicode characters and not numerical ones. In short, it treats them like strings.

There is an easy fix, the Array.Sort method takes a function as a parameter which allows you to define how the comparison elements are treated:

Now the sort order comes out as:

Custom Sort Running Example

Great! Now the numbers are sorting by their numerical value. Now what if you want the undefined values to appear first in the array? This should be an easy change:

The sort order? It comes out exactly as before:

Put Undefined First Example

So what’s going on?

JavaScript Sort Order Specification

If you look at steps 11 and 12 in the sort order steps in the JavaScript (ECMA Script) specification:

If x is undefined, return 1.
If y is undefined, return −1.

JavaScript automatically handles undefined values when comparing elements in an array. There is no way to override this with a custom comparison function, because the undefined check happens before the comparison is called.

There is a way around this predicament, although it’s not ideal. The current comparison function sorts the numbers in ascending order, and automatically forces undefined values to be last. By changing the way the comparison to put the values in descending order, it forces the elements in the array to be in the correct position in regards to one another but in reverse. To fix the reverse order problem, simply call Array.Reverse() and the order comes out the following way:

This solution causes extra work, and with large datasets the amount of extra time could become noticable, but it does work in a pinch. The other solution would be to completely ignore the Array.sort method and write a custom one.

Sort In Reverse

Language Comparison Euler Problem One

Question:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.

Find the sum of all the multiples of 3 or 5 below 1000.

F#

F# Example One

or

F# Example Two

C#

C# Example One

or

C# Example Two

JavaScript

JavaScript Example One

SQL

or

PowerShell

or

Where’s The Scope

When I started programming, it was customary to declare the variables at the top of the method. Many people stated that it was clearer to declare everything at the beginning and in the order it was used (or alphabetically for easier searching), and some languages, such as Delphi (Turbo Pascal) even required it. As the years passed over the past fifteen years, this has been a debated topic.

From my experience, the shift has been towards only declaring a variable just before needing it. In truth this makes a lot of sense, it allows the compiler to enforce scope and allows programmers the luxury of having it not compile when a variable is used when it shouldn’t be. This comes with the drawback that variable declarations become more difficult to find when searching through. If it’s declared at the top, then it’s always in the same place, although in truth modern IDEs alleviate this problem as it’s normally a single key stroke to find its declaration.

So, what is the correct answer between the two styles? As a blanket rule, the answer is: they’re both wrong. Where to declare variables is significantly different than what to name them. Most languages don’t care about what a variable is named. As long as they follow certain rules (don’t start with a number in some, or must start with a $ in others), the compiler/interpreter doesn’t really do much else with it. Where they are declared though can cause significant changes to how the system functions, and most developers forget that between languages this can make a huge difference.
Take C# and JavaScript for example. They look somewhat the same. Are they? No, they aren’t really even close. In C# variables are created at the branch level:

Running Example

Non-running example

In JavaScript they are declared at the function level:

Running Example

The second alert shows “undefined” instead of “I’m Global” even though it’s at the top of the function, because JavaScript moves all variable declaration to be at the top of the method regardless of where they are declared. (Instantiating the variable when it’s declared in the branch doesn’t initialize it until the branch either. Only the declaration is moved to the top of the function).

Believing that all languages work the same, because one can write similar syntax with error can cause major problems. The language that trips a lot of people up and how they should be declared is SQL. Take the following example:

Here’s the output:

Notice how it only prints out 10 even though the counter is being updated each loop? If the @LocalVariable was actually initialized each time the loop executed, it would have counted down to 1 because it would be set back to NULL each time.

What you have to remember, is that even though you wrote it a certain way doesn’t mean that the compiler/interpreter will output the exact same instruction set you thought it would. Assuming a variable will behave a certain way without testing or explicitly setting it, can be the difference between working code, and spending several hours tracking down a problem.