All IQueryables Are Not The Same

On my current project we use an Object Relational Mapping (ORM) tool named LLBL Gen Pro.  LLBL has a really nice LINQ implementation over it which allows for writing queries much more naturally and in a strongly typed environment.  nHibernate has long had a query language that looked similar to SQL named HQL (Hibernate Query Language).  Unlike LINQ, HQL was weakly typed so changes to types wouldn’t result in compile time errors.  Since LINQ is strongly typed this is not an issue.  When we decided on LLBL Gen Pro it was before nHibernate Linq existed.  We liked being able to use LINQ for querying since could leverage the learning from using LINQ to Objects from a syntactic perspective.

The above is why we choose LLBL and we have been please with the LINQ implementation.  When using LINQ, however, it is important to keep in mind who is the daddy of an IQueryable or IEnumerable you are working with. 

I recently came across a scenario like the below where I retrieved a result set  and then further refine the result set.


//--query against IQueryable that is LLBL Entity
var codes = from c in companyQ
                 where c.CompanyID == companyID
                 select c.PromoCodes;


if (codes.Count() > 0)
            {
               //--get the freetrial from result set
                var freeTrial = codes.Where(c => c.CodeType == (int)CodeTypeEnum.FreeTrial).SingleOrDefault();

              //--get list prepaid from result set
                var prePaids = codes.Where(c => c.CodeType == (int)CodeTypeEnum.PrePaid);


      }

What I didn’t expect was that the second refining, which I meant to happen to the result set in memory, actually spawned another query in the database.  Once I realized this it made sense, since the DataProvider for the IQueryable I was working with was the LLBL Gen Pro Implementation, and it is supposed to query against the database.   To do what I intended I converted the result set to a List before doing my refining which allowed the DataProvider to be the DataProvider underpinning LinqToObjects.  This then did what I wanted and queried against the data in memory.


//--query against IQueryable that is LLBL Entity
var codesFromDB = from c in companyQ
                 where c.CompanyID == companyID
                 select c.PromoCodes;

//--convert to list based Queryable so 
//   further refining done in memory

var codes = codesFromDB.ToList().AsQueryable();

if (codes.Count() > 0)
            {
               //--get the freetrial from result set
                var freeTrial = codes.Where(c => c.CodeType == (int)CodeTypeEnum.FreeTrial).SingleOrDefault();

              //--get list prepaid from result set
                var prePaids = codes.Where(c => c.CodeType == (int)CodeTypeEnum.PrePaid);


      }

The lesson I learned here was keep in mind who owns an IQueryable or IEnumerable when using it, or you might end up with some unexpected side effects.

The Rhino Mock Magic Of GetArgumentsForCallsMadeOn

One of the things that occasionally came in handy using Rhino Mocks previous to their AAA linq based syntax was the ability to see if operations on a mock were performed in a particular order. We could very easily set expectations inside and ‘Ordered’ block and if things happened out of order viola! you recieved an exception. Looked like the below:

       [Test]
        public void CompanySaveTest()
        {
            Company comp = new Company();
            comp.Name = "CompanyName";
            comp.Address = "Add1";
            comp.Phone = "222*111*3333";
            comp.User.Name = "Bill";

            int newCompID = 4;
            int newUserID = 2;

            MockRepository repo = new MockRepository();
            IDbManager dbMock = repo.DynamicMock<IDbManager>();

            using (repo.Ordered())
            {
                repo.Expect(dbMock.SaveComany(comp.Name, 0)).Return(newCompID);
                repo.Expect(dbMock.SaveUser(comp.User.Name, newCompID)).Return(newUserID);
                repo.Expect(dbMock.SaveComany(comp.Name, newUserID)).Return(newCompID);

            }
            
            //--Call Method Under Teset
            CompanyManger.SaveNewCompany(comp);

            repo.VerifyAllExpectations();

        }

When we switched to the AAA based syntax there was no way to reproduce the above, at least no obvious way. Luckily we did not require checking order to often, but when we did a co-op on my current project came up with a very effective alternative using the AAA syntax. The method only supports checking the order of operations called against a single mock object, but at least that’s something. Not the full ordered ability of the past, but I’ll take it when it’s handy. At least we can ensure the save calls on the company are in the right order.

The method ‘GetArgumentsForCallsMadeOn’ can be called on a mock after it has been used and it will return a list of object arrays. Each call on the mock will result in a new object[] in the list holding the parameters used in that call. More importantly the object arrays are added as each call on the mock is made, giving us a way to determine if calls were made on this mock in the appropriate order. As simple example looks like the below:

[Test]
        public void CompanySaveTest()
        {
            Company comp = new Company();
            comp.Name = "CompanyName";
            comp.Address = "Add1";
            comp.Phone = "222*111*3333";
            comp.User.Name = "Bill";

            int newCompID = 4;
            int newUserID = 2;

            MockRepository repo = new MockRepository();
            IDbManager dbMock = MockRepository.GenerateMock<IDbManager>();

            dbMock.Expect(db => db.SaveComany(comp.Name)).Repeat.Any.Return(newCompID);
           
            //--Call Method Under Teset
            CompanyManger.SaveNewCompany(comp);

            IList<object[]> args = dbMock.GetArgumentsForCallsMadeOn(db => db.SaveNewComany(Arg<string>.Is.Anything));

            //-- make sure called twice
            Assert.AreEqual(2, args[0].Length);
            
            //--check to make sure called in right order by checking updatingUserIDs of calls
            int updatingUserID = (int) args[0][1];
            Assert.AreEqual(0, updatingUserID);
            Assert.AreEqual(newUserID, updatingUserID);


        }

Now we can check order of calls on the mock object in question. Another nice thing is we can also do checks that were very cumbersume and not very readable in constraints and Arg<> checks in ‘AssertWasCalled’ methods in a much cleaner way.

//--check to make sure called in right order by checking updatingUserIDs of calls
            int updatingUserID = (int) args[0][1];
            Assert.AreEqual(0, updatingUserID);
            Assert.AreEqual(newUserID, updatingUserID);

            Company companySaved = args[0][0] as Company;
            //--check make sure arg is company
            Assert.IsNotNull(companySaved);

            //--now check what ever we like in cleaner way than constraint checking.
            Assert.AreEqual(newUserID, companySaved.LastModifiedUserID);

One thing to remember, however, is that when recording object calls on a mock the references are what it captures. When you do comparisons on reference arguments you are comparing the data as it is on the object at the end of the method. What this means is if you have an object ‘Customer’ and when passed to method ‘UpdateCustomer’ the ‘Name’ property is ‘Larry’ but the property is later changed to ‘Bill’, then the ‘Name’ property will reflect ‘Bill’, it’s state at the end of the method, whenever it is interrogated in an ‘AssertWasCalled’, ‘AssertWasNotCalled’ or ‘GetArgumentsForCallsMadeOn’ regardless of what the property value was when the method was actually called and recorded. This can be a pain when trying to do asserts, but such is life. In these cases you have to do the argument check on an expect previous to the mock being used.

Handling Dependent Expected Results

Recently the question came up as to how to handle creating expected results for a unit test when those results rely on a call to an outside helper function. As an example, in our current project we have a “Name” class that has methods for creating a string that is formattted to be first name then last name and another function to create a string that is last name comma first name. The name class looks a little bit like the below example:

public class Name
{
    public string FirstName
    { get; set; }

    public string LastName
    { get; set; }

    public string GetFirstNameLastName()
    {
        return this.FirstName + " " + this.LastName;
    }

    public string GetLastNameFirstName()
    {
        return this.LastName + ", " + this.FirstName;
    }
   
}

The question that arose was what to do in tests where some expected result had been manipulated by a helper method, such as one of the name formatting methods in our “Name” class. For instance, let say we had a report class that created one row per customer wtih some information about that customer. The first field of the report is supposed to be the customers name formatted as last name comma first name. In the code we’re using the the ‘ GetLastNameFirstName()’ method of the name object to get the string the report class is printing. Let’s say the report has a ‘ReportDataSource’ that takes a Customer object that has ‘Name’ property called customer name and sets a string ‘Name’ as such: (Below is not real just for example sake)

public class ReportDataSource
{
    public string Name;

    public void FillItem(Customer customer)
    {
        this.Name = customer.CustomerName.GetLastNameFirstName();

     }
}

Let’s say we have a unit test for this that looks like the below, the gist of the question is what should we set as the ‘excpectedName’? (Below test would be overkill itself in reality, again just for exlanatory purposes)

 [Test]
public void GetCustomerName()
{
    //--Arrange

    string expectedName = "";

    Customer cust= new Customer();
    cust.CustomerName.FirstName = "first";
    cust.CustomerName.LastName = "Last";
    
    
    ReportDataSource reportData = new ReportDataSource();

    //--Act
    reportData.FillItem(cust);

    //--Assert
    Assert.AreEqual(expectedName, reportData.Name);
}

We came up with three possiblities, we could hardcode the value so we’d have a line that looked like the below:

string expectedName = "Last, First";

Hard coding seemed fine and makes the test very explicit. The worry for some was what happened if we changed the behavior of the ‘GetLastNameFirstName()’ method? Then we would have to go manually change all the places where we’d hard coded what we expected it to do.

I agreed this was a concern, but my thinking was from a testing perspective we weren’t concerned with how the method did it’s work, we wanted to see that exactly that hard coded string was returned in our test given the input we’d provided. Someone pointed out that this wasn’t necessarily the case, we did not really want to test for our hard coded string since we wanted to rely on the Name class, and we did want any changes there to not break out test. The entire reason for centralizing name formatting was so changes in name formatting would only have to be made in one place. I had to agree.

The next thought if we wanted to make sure the Name object was involved was to mock the Name object.  This sounded like a good idea, we want to rely on the Name object and wanted our test not to break if the Name object changes it’s behavior.  However, as I thought about this it started to seem like overkill. To easily mock the Name object we’d have to extract an interface and go through the extra effort of creating and programming a mock in testing to isolate the behavior of a method that just turns around a name?

I could see this being necessary if the method did something that wasn’t repeatable.  Lets say in our method we were creating a GUID, well we’d have no idea what each new GUID would be, we’d have to mock the call to create our GUID if we wanted to test any output that contained the GUID.  Every call to the “GetLastNameFirstName()’ method will produce the exact same output given the same input.   Which lead me to the solution I liked best.

Given that the ‘GetLastNameFirstName()’ method will produce the expected result we want, and if it’s behavior changes we want to expect the new behavior, why not use the method itself to create our expected result?  Something like this:

[Test]
        public void GetCustomerName()
        {
            //--Arrange

           
            Customer cust= new Customer();
            cust.CustomerName.FirstName = "first";
            cust.CustomerName.LastName = "Last";
            
            //-- use the dependent method to create the test data since we don't want
            //   to test the dependent method just want to make sure we're always consistend with it
            string expectedName = cust.CustomerName.GetLastNameFirstName();
            
            ReportDataSource reportData = new ReportDataSource();

            //--Act
            reportData.FillItem(cust);

            //--Assert
            Assert.AreEqual(expectedName, reportData.Name);
        }

Some still preferred the idea of mocking, but this worked best for me. If the method is deterministic and doesn’t rely on anything outside the immediate application domain, why not let the method act as it’s own mock? This allowed us to not be dependent on the behavior of the methods in the Name class as any changes cause changes in our expected results. We also can avoid the extra effort for mocking. I like it!

Unit Testing LINQ To LLBL

In many of the projects I’ve been working on we use an ORM tool called LLBL Gen Pro.  This tool provides a LINQ implementation that allows us to to select objects from our data source using LINQ query syntax.  An interesting obstacle we encountered was how to unit test code that used LINQ queries passed to LLBL.

For unit testing we did not want to actually hit our database, we wanted to isolate the code under test and ensure that it did what we intended.  Our first thought was to mock the IQueryable interface.  However, in the context of a LINQ query this didn’t seem feasible.   When mocking an object usually we provide a  mock implementation as a stand in and have it provide return values when needed and otherwise track the the object being’s mock usage.  It wasn’t immediately obvious how we would program a mocked IQueryable interface or what calls would be made against it during a particular LINQ query.

Instead of providing a mocked IQueryable interface we realized we could simply create an “in memory” test database of sorts for our tests.  Generic lists provide an ‘AsQueryable’ method that return their contents as an IQueryable.  In order to make our select logic testable we have our methods take the IQueryable to select against as a parameter.  When running for real the IQueryable parameter passed to the method is the data object that really hits the database, when under test  a generic list is passed in.  A very simplified example method might look like the below:

public List<SchoolEntity> GetList(SchoolFilter filter, IQueryable<SchoolEntity> schoolQ)
        {

            var result = from school in schoolQ
                         where school.Name == filter.Name
                                 && school.Colors == filter.Colors
                         select school;

            return result.ToList();

        }

In order to test we can create a list to pass in to our method:

[Test]
 public void TestSchoolColorFilter()
 {
 //--Arrange
 SchoolFilter filter = new SchoolFilter();
 filter.Colors = "Blue";

 List<SchoolEntity> schools = new List<SchoolEntity>();

 SchoolEntity school1 = new SchoolEntity();
 school1.Colors = "Blue";

 schools.Add(school1);

 SchoolEntity school2 = new SchoolEntity();
 school2.Colors = "Purple";

 //--Act
 List<SchoolEntity> retVal = GetList(filter, schools.AsQueryable());

 //--Assert
 Assert.AreEqual(1, retVal.Count());

 }

When actually running the method is called after a reference to an IQueryable object from the database is created and passed in like the below:

public List<SchoolEntity> GetList(SchoolFilter filter)
 {
         using (DataAdapter data = new DataAdapter())
         {
            LinqMetaData meta = new LinqMetaData(data);
            return this.GetList(filter, meta.School);

         }

 }

This allows us to take advantage of LLBL’s LINQ to LLBL functionality but still create unit tests that don’t have to cross the boundary to the database in order to run. So far it’s helped us to keep our code less coupled and more cohesive. The next hurde, and another post, is how we handle unit testing when LLBL specific LINQ extension methods are involved, aka Prefetch Paths!

Using LINQ Expressions For Dynamic Predicates

I ‘ve recently been building a set of reports that all output data in a common format, and also group their data in a common format.   Each report needs to be able to have a date predicate based on day, week or month.    Using LINQ to LLBL Gen Pro (the ORM tool I’m currently using) I set up a base report class that has abstract methods defining the date predicates and query bodies, but share the grouping and output mechanisms.  What allows this is being able to return a LINQ Expression in the methods that define the date predicates.

The abstract date predicates end up looking like the below:

protected abstract Expression<Func<T, bool>> GetDayDatePredicate(int reportBuckets);

Once implemented the date predicate ends up being something like:

protected overrides Expression<Func<dataEntity>,bool>> GetDayDatePredicate(int reportBuckets)

{

return d => d.CreatedDate.Date > DateTime.Now.AddDays(-1 * reportBuckets).Date;

}

Having the predicates in method form serves two purposes: first the method name clearly states what the predicate is doing, (fairly straightforward above but bucketing by weeks is not nearly as clear), and second it allows the base class to create the query by doing something like the below:

public void RunQuery(int numberOfBuckets, IQueryable<T> q)
{

Expression<Func<T, bool>> datePred = GetDayDatePredicate(numberOfBuckets);
q = q.where(datePred);
RunQuery(q);

}

This allows a base class to manage running several reports if they all have similar structure.  When adding a report I just add a new implementation of the base class and fill in the abstract bits and then I know I have a report that our UI can use, since it relies on the common output the base class enforces on its children.  Don’t know if it’s all that good a solution, we’ll see.