API Design: Batch Operations

When designing a batch API, it is important that the return results be relatable to the individual arguments that were asked to be processed.

Consider an API that takes a list of objects to perform an operation on. It might have a method signature that looks like this:

        Response Process(Record[] records);

 

What is a good design for the Response object? I recently encountered a situation in which I was given a response object like this:

    public class Response
    {
        public bool Success { get; set; }
    }

 

Okay, actually it was more like this:

        bool ProcessTheRecords(int[] ids);

 

The problem with this approach is that when the operation fails there is nothing on the Response to indicate the nature of the failure. A better Response object might look like this:

    public class Response
    {
        public bool Success { get; set; }

        public OperationResult[] ErrorsAndWarnings { get; set; }

        public ProcessRecordResult[] Results { get; set; }
    }

    public class ProcessRecordResult
    {
        public string RecordIdentifier { get; set; }

        public OperationResult[] ErrorsAndWarnings { get; set; }
    }

    public class OperationResult
    {
        public string Message { get; set; }
        public Severity Severity { get; set; }
    }

    public enum Severity
    {
        Error = 1,
        Warning =2,
    }

Why is this better? Response.ErrorsAndWarnings lets me see if anything went wrong with the operation outside of any particular record. There should be a ProcessRecordResult for each record passed into the method call. Each ProcessRecordResult has its own list of ErrorsAndWarnings that indicates what may or may not be wrong with a specific record.

The purpose of the response object is to give enough information to the caller so that they can adjust their failed request into a successful one. Minimize the size and shape of the data required to perform the operation. Maximize the size and shape of the data required to understand what went wrong during the operation. These principles apply to non-batch operations as well, but they are especially frustrating when there’s an error in one of hundreds of records.

Simple.Validation 0.5.0 Released

New Feature

Message() method has been overloaded to accept a function to build the validation message while the object/property is being validated.

Example:

var validator = Properties<Employee>
    .For(e => e.FirstName)
    .Required()
    .Message((context, value) =>
        {
             customMessage = string.Format("Custom Message format '{0}'", value);
             return customMessage;
        });
Origin Story: How I Became a Software Developer

Scott Hanselman recently posted his story about how he became a software developer. This is mine.

I was in debt. And I needed money.

When I was in my early 20’s I didn’t have much going for me. I had quit high school in the 10th grade and mainly worked in the fast food industry. I didn’t have much money, but I wanted stuff—so I use credit cards heavily. I got in over my head really fast.

I had been toying with school. I was half-assedly pursuing a philosophy degree, but I wasn’t really putting the effort into that that I should have been. I was unhappy, alone, and going nowhere fast.

At 25 I had a realization—if I met the kind of woman I wanted right then, she would want to have absolutely nothing to do with me. It was a slap in the face. I had this high standard that I expected my future romantic partner to live up to, but I had done nothing to merit her attention. I was a loser.This was my way of judging myself by my own standards. Something had to change.

I had recently seen an article in the newspaper about a high school kid making $50k/year building websites in his spare time. This amount of money seemed unfathomable to me. With that kind of money, I could pay my debts and be on my way. This encouraged me to take some programming classes at the local community college. I had to make some arrangements with my employer so that I could leave work in the middle of the day for classes. I would get to work at 8AM, leave for class around noon, and come back and work until midnight or so. It was hard.

I struggled with nested for-loops in the bubble sort algorithm, but I had a knack for organizing my code in a readable fashion. I managed to get through QBasic, C, and Visual Basic before I decided to submit my resume at the school’s job board.

I got lucky.

It happened that a local company was looking for someone that they wouldn’t have to pay very much and that they could train to work on their software. I had a message on my answering machine from Benny when I got home from school that day.

We scheduled the interview and I showed up with my code-sample—a windows form with a set of radio buttons that toggled an image of a flag for an associated country. I knew it wasn’t much, but I wanted to show that I could do something. I think Benny barely looked at it. The interview actually went well. Benny gave me an overview of the application we would be working on. Their customers were the fast-food industry so my experience there would help a little bit. At least I understood the business domain. After a little probing about my skills, we started talking about “South Park” for another 20 minutes or so. A friendship was born.

Apparently they were desperate. One of my competitors for the job had picked a fight with the secretary before the interview even started! Benny called me a couple of weeks later to tell me that I got the job. I was desperate too. The money wasn’t much—in fact it was a pay cut for me since I would be salaried without overtime compensation. I took the job.

Working with Benny was a revelation. He knew that I didn’t know much, but he also knew that I had a brain in my head and a willingness to learn. He was patient with me while I applied myself and learned to be an effective contributor to our application. My skills grew over time and inside a year I could add to the system almost as easily as he could.

It was 5 years before I managed to pay off my debt. In that time I had learned much about software algorithms, patterns, and practices. I had started my first steps down the road of agile software development practices. I was reading everything I could get my hands on about better ways of writing and managing software. I was evangelizing what I had learned to my co-workers. I started down this path to find a way to pay my bills without filing for bankruptcy. What I found was a passion for a profession that I’m actually pretty good at, and a great new friend.

Benny and I are still friends to this day. We’ve moved on to different jobs, and I moved 3000 miles away to Seattle (I’m hoping to convince him and his wife to move out here!) I shudder to think what might have happened to me had I not been lucky enough to get my start in this field. Getting my foot in the door with him is one of the greatest things that’s ever happened to me.

Thanks Benny.

Isg.EntityFramework 0.5.1 – Release Notes

I’ve added a LookUp property to the InterceptionContext to make it easier to see which rows were affected during the After() interception phase.

    public class InterceptionContext
    {
        public DbContextBase DbContext { get; internal set; }
        public ObjectContext ObjectContext { get; internal set; }
        public ObjectStateManager ObjectStateManager { get; internal set; }
        public DbChangeTracker ChangeTracker { get; internal set; }
        public IEnumerable<DbEntityEntry> Entries { get; internal set; }
        public ILookup<EntityState, DbEntityEntry> EntriesByState { get; internal set; }

... snipped for brevity

EntriesByState is populated prior to the call to Before(). Added and Modified entities will have their EntityState reverted to UnChanged after SaveChanges() is called. EntriesByState preserves the original state of the entities so that After() interceptors can make use of new Id’s and such.

Simple.Validation 0.4.0–Release Notes

New Features

Added an assertion api to the PropertyValidator so that the Properties<T> api can be extended by clients.

The order of operations for the Properties api is as follows: If(), Required(), Assertions().

As an example, customized support for boolean properties was added to the library using the following code:

    public static class BooleanPropertyExtensions
    {
        public static PropertyValidator<TContext, bool> IsTrue<TContext>(this PropertyValidator<TContext, bool>  self)
        {
            self.Assert((t, p) => p);
            return self;
        }

        public static PropertyValidator<TContext, bool> IsFalse<TContext>(this PropertyValidator<TContext, bool> self)
        {
            self.Assert((t, p) => !p);
            return self;
        }

        public static PropertyValidator<TContext, bool?> IsTrue<TContext>(this PropertyValidator<TContext, bool?> self)
        {
            self.Assert((t, p) => p.GetValueOrDefault());
            return self;
        }

        public static PropertyValidator<TContext, bool?> IsFalse<TContext>(this PropertyValidator<TContext, bool?> self)
        {
            self.Assert((t, p) => p.HasValue && !p.Value);
            return self;
        } 
    }

 

Breaking Changes

RangePropertyValidator has been removed. It’s functionality is still available, but it has been entirely replaced by extension methods that use the new Assert() method on the PropertyValidator.

ValidationResultTypes have been completely removed. Then intent for ValidationResult.Type is that it is custom per project.

What’s new in Simple.Validation 0.3.1?

Silverlight 4

The entire Simple.Validation library has been compiled for Silverlight and included in the NuGet package.

Conditional Property Validators

An If() method has been applied to the following property validators:

When If() is called with the required Predicate, the validator will only apply when the condition specified by the Predicate is met.

        [Test]
        public void If_PredicateIsFalse_ShouldNotValidate()
        {
            // Arrange
            var validator = Properties<Employee>
                .For(e => e.Age)
                .GreaterThanOrEqualTo(18)
                .If(e => e.Age != -1)
                ;

            // Act
            var employee = new Employee()
            {
                Age = -1
            };
            var results = validator.Validate(employee);

            // Assert
            Assert.That(results, Is.Empty);

        }
Simple.Validation v0.2.3

A new release of Simple.Validation is available on NuGet.

The changes include:

While Simple.Data is not intended to focus on property-level validation so much that more complex validation scenarios are difficult, it must be recognized that property-level validation accounts for a large part of validation scenarios. In keeping with the goal of simplicity, Simple.Validation provides some mechanisms for wiring up property-level validations quickly. The Fluent Properties API provides factory methods for creating validators for common property types. When used in conjunction with the CompositeValidator<T> it is a trivial task to get simple property-level validation implemented quickly.

This code is from the Personnel.Sample project included in the source code.

    public class SaveAddressValidator : CompositeValidator<Address>
    {
        public override bool AppliesTo(string rulesSet)
        {
            return rulesSet == RulesSets.Crud.Save;
        }

        protected override IEnumerable<IValidator<Address>> GetInternalValidators()
        {
            yield return Properties<Address>.
                For(a => a.Line1)
                .Length(0, 50)
                .Required()
                .Message("Address Line 1 is required.");

            yield return Properties<Address>
                .For(a => a.Line2)
                .Length(0, 50)
                .NotRequired()
                .Message("Line 2 must be between 0 and 50 characters in length.");

            yield return Properties<Address>
                .For(a => a.Line3)
                .Length(0, 50)
                .NotRequired()
                .Message("Line 3 must be between 0 and 50 characters in length.");

            yield return Properties<Address>
                .For(a => a.PostalCode)
                .Length(0, 20)
                .NotRequired()
                .Message("Postal Code must be between 0 and 20 characters in length.");

            yield return Properties<Address>
                .For(a => a.Country)
                .Length(0, 3)
                .NotRequired()
                .Message("Country must be between 0 and 3 characters in length.");

            yield return Properties<Address>
                .For(a => a.StateOrProvince)
                .Length(0, 50)
                .NotRequired()
                .Message("StateOrProvince must be between 0 and 50 characters in length.");

        } 
    }

 

In this sample, each call to Properties<T>.For() returns an instance of StringPropertyValidator. CompositeValidator will accumulate the results from each of the property-level validators and return them all as a single result set. What about non-string properties?

    public class SaveEmployeeValidator : CompositeValidator<Employee>
    {
        public override bool AppliesTo(string rulesSet)
        {
            return rulesSet == RulesSets.Crud.Save;
        }

        protected override IEnumerable<IValidator<Employee>> GetInternalValidators()
        {
            yield return Properties<Employee>
                .For(e => e.FirstName)
                .Length(3, 50)
                .Required()
                .IgnoreWhiteSpace();

            yield return Properties<Employee>
                .For(e => e.LastName)
                .Length(3, 50)
                .Required()
                .IgnoreWhiteSpace()
                ;

            yield return Properties<Employee>
                .For(e => e.Age)
                .MinValue(18)
                .MaxValue(65)
                ;

            yield return Properties<Employee>
                .For(e => e.Address)
                .Required()
                .Cascade("Save")
                ;

            yield return Properties<Employee>
                .For(e => e.ContactInfo)
                .Required()
                .Count(1)
                .Unique<ContactInfo>(c => c.Type)
                .Cascade("Save");

        }
    }

Employee.Age is of type Int32. Properties<T>.For() any property that is convertible to IComparable will return an instance of RangePropertyValidator. If the property is a reference type, it will return an instance of ReferencePropertyValidator, and if it’s a collection type it will return an instance of EnumerablePropertyValidator.

ReferencePropertyValidator and EnumerablePropertyValidator are similar in that they support Cascading validation onto the reference or collection property itself. This means that Valdiator.Validate<T> will be called for the value of the reference property, or for each element of the collection property. The results will be accumulated into the overall validator results.

When performing cascade validation, there is some question about the type that should be passed to Validator.Validate<T>. Should it be the declared property type (or declared type of the elements of the collection), or should Simple.Validation use the actual type of the property value (or collection element). By default, Cascade() uses the actual type of the property value or collection element, but you can specify the usage of a base type or interface by using the Cascade<T> overload.

In the Personnel.Sample project EmailAddress inherits ContactInfo. There is an EmailAddressValidator that tests the email address against a regular expression. If an instance of EmailAddress is added to the Employee.ContactInfo collection, should it be validated using the EmailAddressValidator or the ContactInfoValidator? Calling the non-generic Cascade() method will cause it to be validated using EmailAddressValdiator. Calling Cascade<ContactInfo>() will cause it to be validated only by the SaveContactInfoValidator.

Announcing Simple.Validation

My company has graciously allowed me to develop a lightweight validation framework and publish it as open-source. There are other very nice validation frameworks available, so why another one? I wanted to be able to use something with all of the strengths and none of the weaknesses of the existing libraries. Here are my stated goals for the project:

  • Simplicity
  • Get moving with a minimum number of moving parts
  • Support dependency injection
  • Focus on object-level validation
  • Support for common property-level validation
  • Support all application types in a consistent manner
  • Support validation at all application layers in a consistent manner
  • Pluggable Architecture

The api I wanted for usage was very simple. Here’s some code from the LoanApplicationSample project:

    public class LoanApplicationDb
    {
        public void Save(LoanApplication loanApplication)
        {
            Validator.Enforce(loanApplication, "Save");
            // do save logic
        }

        public void Submit(LoanApplication loanApplication)
        {
            Validator.Enforce(loanApplication, "Save", "Submit");
            // do submit logic
        }
    }

Just pass your object to the Valdiator.Enforce() method along with the rules sets you wish to validate against and you are off to the races. Enforce will call Validator.Validate(), check for errors, and throw a ValidationException if it finds any. If it doesn’t find any, it just returns the results of Validate(). If you want to bypass the exception logic, just call Validate() directory.

The above code sample demonstrates how to consume the Validator API, but how do you plug into it? You need to be aware of two components. The first is the IValidator<T> implementation. Here is a sample validator that validates the LoanApplciation against the “Save” rules set.

    public class SaveLoanApplicationValidator : IValidator<LoanApplication>
    {
        public bool AppliesTo(string rulesSet)
        {
            return rulesSet == "Save";
        }

        public IEnumerable<ValidationResult> Validate(LoanApplication value)
        {
            var nameResults = Properties<LoanApplication>
                .For(e => e.Name)
                .Required()
                .Length(2, 100)
                .Message("Name is required.")
                .IgnoreWhiteSpace()
                .Validate(value)
                ;

            var reasonResults = Properties<LoanApplication>
                .For(e => e.Reason)
                .Required()
                .Length(10, 500)
                .IgnoreWhiteSpace()
                .Message("Reason is required.")
                .Validate(value)
                ;


            var accumulatedResults = nameResults
                .Concat(reasonResults)
            ;

            return accumulatedResults;
        }
    }

 

This validator uses the built-in PropertyValidator fluent syntax. Use of the built-in validators is entirely optional. They are included as a convenience in order to provide support for common property-level validations. Simple.Validation handles the simple stuff for you Smile. Where many validation frameworks fall down is when there is some kind of complex validation. In Simple.Validation, complex scenarios are handled the same way as property-level validation. Here’s an example:

        public IEnumerable<ValidationResult> Validate(LoanApplication value)
        {
            if (someComplexCondition)
                yield return new ValidationResult()
                                 {
                                     Context = value,
                                     Message = string.Format(messageFormat, arguments),
                                     PropertyName = "PropertyName",
                                     Severity = ValidationResultSeverity.Error,
                                     Type = MyCustomValidationType.SomeComplexConditionError
                                 };

The ValidationResult is a fairly rich object containing a lot of information about the validation error. The Context is the object being validated. You can (and should) provide a detailed message about the error. PropertyName is not required, but will be useful in many scenarios. Severity has levels of Error, Warning, and Informational. ValidationResult.Type is a property there for arbitrary categorization of validation results.

Okay, so now I’ve written a validator—but how does the static Validator class know about the validator I just wrote?

Validator uses an instance of IValidatorProvider to get the list of validators from the simple. Simple.Validation ships with a DefaultValidatorProvider which you can use as follows:

    public class Configuration
    {
        public void ConfigureValidation()
        {
            var validatorProvider = CreateValidatorProvider();
            RegisterValidators(validatorProvider);
            Validator.SetValidatorProvider(validatorProvider);
        }

        private static void RegisterValidators(DefaultValidatorProvider validatorProvider)
        {
            validatorProvider.RegisterValidator(new SaveLoanApplicationValidator());
            validatorProvider.RegisterValidator(new SubmitLoanApplicationValidator());
        }

        private static DefaultValidatorProvider CreateValidatorProvider()
        {
            var validatorProvider = new DefaultValidatorProvider();
            return validatorProvider;
        }
    }

The implementation of DefaultValidatorProvider is naïve, so you are encouraged to use the IValidatorProvider interface as a wrapper around your favorite Dependency Injection framework. We provide a second nuget package called Simple.Validation.Ninject that contains a NinjectValidatorProvider.

We’ve been using Simple.Valdiation internally for awhile and feel pretty good about its release. However, as it hasn’t been published on the public NuGet feed yet, we’re assigning a prelease version of 0.1. We will conform to semantic versioning conventions as outlined in this post from Phil Haack.

So, run Install-Package Simple.Validation from the Visual Studio Package Manager Console and start using the bits! As always, your feedback is much appreciated.

Happy Coding! Open-mouthed smile

Isg.Specification

I was reading some interesting blogs on the specification pattern today. I was hoping to find a NuGet package, but no such luck. So, I put together the best ideas from the blogs I read and created a new NuGet package called Isg.Specification. Much of this work is not original—my contribution is putting the work of these other developers together in an integrated way and publishing it on NuGet.

Although the write-up of the specification pattern on wikipedia doesn’t mention my specific usage goal, I want to create a parameter object that provides a specification to be used by a LINQ query in EntityFramework. In this way, I can provide a single Get<T>(ParameterObject specification) method to service all Get queries. This simplifies a service interface by removing all the various GetById, GetByName, GetByHasInvoices methods. All that differs between those methods is the predicate applied to the query, so passing the predicate into Get() as a parameter just feels natural. However, I don’t want the caller to be able to specify any predicate they want so using a parameter object that converts itself into a predicate services the purpose of limiting the callers query access as well as simplifying the query API.

Even though my target usage is EntityFramework, and the source code is in the same repository as Isg.EntityFramework, Isg.Specification in no way depends on EntityFramework.

The simplest form of the specification pattern uses simple boolean functions. However, since my target usage is EntityFramework, I needed the functions to be convertible to SQL by the EntityFramework. For this reason, my implementation of ISpecification<T> uses Expression<Func<T, bool>> instead of Func<T, bool> as its signature.

   1:      public interface ISpecification<T>
   2:      {
   3:          Expression<Func<T, bool>> IsSatisfied();
   4:      }

 

The simplest way to get started using the library is to inherit from CompositeSpecification<T>. Using the same sample DbContext I used for Isg.EntityFramework, here is a sample implementation of a Specification object against Customer:

   1:      public class CustomerSpecification : CompositeSpecification<Customer>
   2:      {
   3:          public string Name { get; set; }
   4:          public bool? HasInvoices { get; set; }
   5:   
   6:          protected override IEnumerable<Expression<Func<Customer, bool>>> GetExpressions()
   7:          {
   8:              if (!string.IsNullOrWhiteSpace(Name))
   9:                  yield return ByName();
  10:   
  11:              if (HasInvoices.HasValue)
  12:                  yield return ByHasInvoices();
  13:          }
  14:   
  15:          private Expression<Func<Customer, bool>> ByHasInvoices()
  16:          {
  17:              if (HasInvoices.Value)
  18:                  return c => c.Invoices.Any();
  19:              return c => !c.Invoices.Any();
  20:          }
  21:   
  22:          private Expression<Func<Customer, bool>> ByName()
  23:          {
  24:              return c => c.Name.StartsWith(Name);
  25:          }
  26:      }

 

Usage is pretty easy. ISpecification<T> exposes a single method called IsSatisfied(). Feed the result of this method to a LINQ Where clause and you’re off to the races. CompositeSpecification<T> aggregates the expressions provided by GetExpressions() into a single And-ed expression and uses that for the filter.

In my test cases I create 2 customers, one of whom has invoices and one who does not. You can review the test cases in full here:

Here is sample usage:

   1:          [Test]
   2:          public void HasInvoices()
   3:          {
   4:              // Arrange
   5:              var filter = new CustomerSpecification()
   6:                               {
   7:                                   HasInvoices = true,
   8:                               };
   9:   
  10:              // Act
  11:              using (var context = new CustomerDbContext())
  12:              {
  13:                  var query = context.Customers
  14:                      .Where(filter.IsSatisfied())
  15:                      .ToList();
  16:   
  17:                  // Assert
  18:                  Assert.That(query, Has.Count.EqualTo(1));
  19:                  var result = query.Single();
  20:                  Assert.That(result.Name, Is.EqualTo("Dagny Taggart"));
  21:              }
  22:          }
 
I’ve published Isg.Specification as version 0.1. As always, if you find use for my libraries I would really appreciate your feedback.
 
Enjoy!
Interesting Blogs on the Specification Pattern

I just wanted to give some link-love to the blogs that were helpful to me in implementing the Specification Pattern. Stay tuned for Isg.Specifications on NuGet!

http://davedewinter.com/2009/05/31/linq-expression-trees-and-the-specification-pattern/

http://blogs.msdn.com/b/meek/archive/2008/05/02/linq-to-entities-combining-predicates.aspx

http://iainjmitchell.com/blog/?p=550

http://huyrua.wordpress.com/2010/07/13/entity-framework-4-poco-repository-and-specification-pattern/

http://devlicio.us/blogs/jeff_perrin/archive/2006/12/13/the-specification-pattern.aspx

Previous Page · Next Page