by Michael Figura
October 16, 2018
Halloween is a few weeks away and everyone loves a good scare, except when it comes to enterprise-integration projects. Even a small surprise can ruin your entire work week. From project management calamities to architectural mutiny, we all head to the office knowing our next fright-fest could be right around the corner. It's important to prepare for these incidents in advance so, when they occur, you and your team can react swiftly and confidently. That’s why, this month, the team at OAGi has prepared a few tips and techniques we’ve collected over the years to help keep you and your project safe and sound.
Let's discuss a few real-world examples from the trenches of enterprise-integration projects. These are tales of true accounts we've experienced in the field on various projects over the years.
The Point-to-Point Sneak Attack
You’d be surprised how often this occurrence can blindside you. Imagine you’re on an integration project where a system like SAP is sending transactions to a system like PeopleSoft and vice versa. While out to lunch with the team, they mention some changes coming next month when they upgrade to the latest version of PeopleSoft. Namely, the fact that they won’t need to use your canonical message types anymore because PeopleSoft now has a built-in SAP adapter. The dev lead might say “They can talk directly to each other now, so why do we need this middle layer?”
The Case for Canonical
The point-to-point argument is tough to refute. On one hand, it makes perfect sense that an organization would keep the integration layer as simple as possible at first. This is the “just get it done” attitude. But, this approach lacks long-term architectural vision. And, on large-scale projects, it can sometimes seem to the architects and those with an architectural vision that the inmates are suddenly running the asylum.
It’s important that you and your team can quickly and accurately point out why this approach would be a huge detriment to the project architecture long-term. On a small scale, this point-to-point approach might seem simpler and easier, but on a large scale it quickly turns into spaghetti integration. Figure 1 illustrates the difference.
A simple illustration isn’t enough to justify major architectural decisions, so let’s back up Figure 1 with concrete examples and numbers. The mathematics of scaling up a many-to-many integration layer are straightforward enough that a basic equation can be applied. The number of possible connections among any number of items is n * (n - 1) for two-way connections, which is described in Figure 2.
Now let’s apply the math to a many-to-one style architecture that makes use of canonical schemas. Scaling this architecture is now much more manageable. Figure 3 below shows this approach, but the math equation has changed a bit. The number of possible connections among any number is now represented by the equation n * 2, and thus the number of required FTEs is far less when the number of integration points grows large.
The Agile Mapping Spec Challenge
Various Agile project methodologies have been on the rise for over two decades now and the benefits, such as leaner team structure and faster time to market, are quickly being realized on projects all over the world. Traditionally, integration-layer projects have followed a Waterfall approach to goals, implementation, and deadlines. But like every other aspect of software development life-cycle, the integration layer is experiencing digital transformation, as well. So, what happens when the mapping developers who write code against a mapping specification only receive a small portion of the fields that need to be mapped? If this dilemma doesn’t raise a brow, wait until you see the look on the mapping developer’s faces when you hand them a Bill of Materials mapping spec with 5 field mappings defined, and tell them there are 40 more coming soon. This can aggravate your dev team, causing frustration and a lack of confidence in the project. A lot of mapping experts might ask “Why should I open up and edit my mapping code 8+ times when I can wait until you’ve got your requirements straight and then write the code in a single sitting?”
Taming the Agile Sprint Schedule for Mapping Developers
Selling the Agile methodology to old-school map developers can be a daunting task, as they usually expect the mapping specs to be around 85% accuracy before being asked to write code. This approach to translation work exists for a few reasons but, it usually occurs because of an attachment to Waterfall. Integration projects have been predominantly Waterfall-style projects for the past 40 years and, while Agile SCRUM practices are quickly being adopted, the mapping developer work often follows this older implementation mentality. Here are a few points to help your case in selling your developers on the Agile process:
- Agile iterations are great for large projects – Large projects that involve multiple departments within the organization benefit greatly from the iterative nature of Agile. Small sprint iterations allow work and progress to be performed without trying to answer every single question and challenge upfront.
- Agile delivers fewer bugs – Since Agile iterations allow developers to code and test smaller units of work during each sprint phase, the quality of delivery goes up. If the QA phase is implemented correctly, every sprint phase set of deliverables should end up being tested multiple times, as every test case is run over again during each end-of-sprint QA session.
- Faster deliverables and shorter time to market – On a good week, at end-of-sprint, the team should be grinning from ear to ear at how much they’ve delivered and tested in such short time. While this doesn’t always happen, when it does it’s a very positive thing for both the team morale and the executive summary. Agile development eliminates fear and uncertainty and reinforces positive feedback…when done properly.
- Agile makes for happier developers – Rather than constantly worrying about large deadlines and deliverables, Agile breaks units of work into smaller, more manageable targets that developers can live with. The stress level should drop significantly in comparison to the last 2 months of a Waterfall project, and expectations are continuously revisited at the end of every sprint. As always, if implemented correctly, an Agile developer is a happy developer.
Mapping Specification Nightmares
At OAGi, our experts have seen every variation of mapping specification you can imagine. In some serious nightmare cases, there is no mapping spec at all. And in most cases, unfortunately, mapping specs are generally handed to the development team by analysts or other departments, and the folks who wrote them are usually not mapping developers themselves. Why is this a problem? Because it makes for nasty, hard-to-use specs. Nothing can aggravate developers more than poorly organized specs. From bad color schemes, to the infamous “pregnant” specs which scroll mostly left to right, there are countless mistakes anyone can make. Here are a few things you want to avoid:
- Let the developers drive the format – Every developer works differently. It’s best to let the developer tell the analysts and mapping requirement folks what format they prefer their content in. We have a great example of a mapping spec template below.
- Keep it simple – Make sure your specs stay simple. There should be a source field, a destination field, any logic or commentary the developer needs to be aware of, such as formatting or cross-walk table lookup, and anything that might help with iterative logic, such as XPath logic. That’s it. Nothing more. Don’t overuse Excel’s row and cell functionality to send your object hierarchy spiraling endlessly downward left to right in an unreadable format.
- Keep your specs scrolling top to bottom – With a little attention and detail, you can create a spec that doesn’t scroll left to right. Why is left-to-right scrolling a hassle for developers? Because mapping developers are almost always hot-keying between windows, and the less they touch the mouse to scroll left and right, the happier they’ll be.
Achieving Better Mapping Specifications That Support Agile
I’ve already mentioned the importance of using great developer mapping specs. Believe it or not, it’s not as difficult to achieve this type of spec as you might think. In fact, this Halloween season, the OAGi team is offering a free gift: the OAGi Developer Mapping Specification Template, which follows all the rules listed above. You’ll even notice there’s a column called “Validated” to better support the Agile dev process. If the validated column contains a Y, it’s ready for the developer to map. And you’ll also notice it doesn’t scroll much left to right. In fact, you can collapse the rows so that you can focus on only the field mappings you’re currently working on. This example maps SAP tables and fields to an OAGIS ProcessBOM BOD document. This is one of the most efficient mapping specs we’ve ever created, and we’re proud to finally share it with you. Happy mapping!
