I just got my weekly issue of (fill in your weekly computer rag) and under the news section is “Difficult ERP Rollout Slows Furniture Maker”. Is it just me or does this seem like the norm with SAP R/3 rollouts?
Let's see there was
The City of San Antonio
Rowe Furniture
Dell Computer
That whole UK fiasco with EDS ( I pretty sure that was SAP R/3 )
So my question is who is buying this crap and why do companies continue to try and roll it out when there is one high profile, expensive, failure after another?
Dell Computer tried to use SAP to replace its DOMs (Dell Order Management System). This was many years ago and the price tag that I recall was around 14 million dollars. For what? Not a single Dell employee was using SAP when I worked there. In fact Dell tried to replace DOMs so many times it became something of a joke [but that's another post]. It is always amusing to watch a company squirm because their entire backend system is running on a competitors hardware (Tandem).
Has anyone worked on a SAP R/3 rollout that wasn't a death march? I really want to know. Five or six years ago I was seriously considering jumping to the SAP platform because the developers were making gobs of money. These days you couldn't pay me to work on a SAP rollout. I'm both older and smarter. Who wants to work on something doomed to fail?
I've been spending a lot of quality time with Team System creating, running, and cursing web tests and load tests. Don't get me wrong here, the product is a lot better than trying to write load tests in ACT, but it falls short in a number of places. It's still beta so hopefully these things will be addressed before RTM.
My Current List of Issues
No way to share test context between tests -- I want to be able to create a test called LogIn that only logs a user in. Then I want to create a test that is ModifyAccount that only modifies the user account. The ModifyAccount test depends on the user being logged in so I want to be able to point it to LogIn and have that context (cookies, etc) pass through. Otherwise I have to duplicate the LogIn code in each and every test that requires a user be authenticated. That sucks!
Memory Usage -- I'm running with 1Gig of RAM and the default selection of counters causes the test to fail due to lack of memory in minutes. In fact I had to close everything down and remove well over 50% of the counters in order to get a test to run for more than a few minutes. On top of that if I used SQLExpress to store the results it's memory usage would grow as long as the test was running (which would only be about 5 minutes) and again it would shut down. I had to change to Xml storage to get a test to run for any length of time.
DataSources -- The whole datasource concept needs some work. Once you setup a datasource there is no way to change it. For example I changed a table name. The datasource contains the table names and doesn't allow you to change them once it is created. In fact the entire database could be missing, but the datasource will happily act as if it is available. Another issue I ran into was the fact that I had two tables, one of which was empty [no rows] and the other had data. My webtest was bound to the table with data however the test refused to run because it said that it was bound to a table with no rows [which it wasn't using]. Since I can't modify the tables I had to insert a dummy record into the empty table in order to get the test to run.
Graph View -- The graph view of the performance counters is nice, but the decision to vary the scale dynamically seems like a poor one especially from a Human Factors point of view. Now, when looking at a running test, or test results, I can't compare the lines I have to mentally do the coversion to make sure everything has the same scale. Also I noticed if you add two counters that are similar but vary only by page name only the first counter will be added to the graph. Evidently the graph decides the counter is already there so it doesn't show it again.
Location of Results -- One of the issues I'm having is that I'm running out of space on my C drive. The XmlResults files are placed in C and I couldn't find a dialog that would allow me to move them somewhere else. This is a real problem because the test will shut down if you run out of space. I should be able to have these files live anywhere, not just under some folder in Documents and Settings
Test Settings -- Another more general complaint is that after you have created your load test and you decide you want to change some values (number of users, think time, whatever) it is no small feat to figure out where to go. There are many items in the load test tree and except for a few of the more obvious ones finding a specific entry/value is a hassle. The opening wizard is nice, so about a single click relaunch of the wizard to keep it that way?
A couple of replies to my recent posts on ASP.NET Context and using the Strategy pattern have really got me thinking about one of the design guidelines I've been following. Tim Haughton posted an example of the strategy pattern and posted that I should consider NMock to handle my ASP.NET Context testing.
Now in the past I have looked at NMock and for a lot of things it makes great sense, but it tends to be a lot of work and the benefit to work ratio isn't necessarily positive (in my past experience). So I have not gone down that route.
Both Tim Haughton and Michael Polo either directly or indirectly were saying use Dependency Injection to handle these issues (which is what NMock does). In the past I've always considered writing my code so that it can be tested to be less than ideal. In fact if I had to write an interface specifically for a unit test and for no other reason I wouldn't do it because it just didn't “feel” right.
I've mostly decided that I was wrong. The extra effort to design for depenedency injection scenarios, especially in code that is hard to test, [e.g. ASP.NET Context] is not only smart, in the end it would be more efficient. It is far more important to have code that is testable than to have some obscure purity of design.
So thanks Tim and Michael for helping me see beyond the blinders I had put on!