Mastering ArchiMate

The ArchiMate 3.0 Relations Table…

…has serious, and many, errors. According to my personal analysis, 152 valid relations are missing, and potentially 2941 relations that are in that table shouldn’t be, if the text of the standard is to be believed or if we use some common sense. Here is why I believe that, and how I did calculate the table myself (code included).

The ArchiMate standard’s metamodel consists of:

  1. A set of element types (e.g. Business Object, Node, Assessment, Capability) in a number of domains (Core, Motivation, Strategy, Implementation)
  2. A set of relation types (e.g. Access, Serving, Realization, Assignment)
  3. A core set of relations defined from element to element (e.g. Application Function — Accesses — Data Object, Technology Interface — Realizes — Requirement)
  4. A set of derived relations (more about these below)
  5. Relations to/from relations (as of ArchiMate 3)

We ignore 5 for now.

3 and 4 together fill the the table in appendix B of the standard. Formally, this table defines all the allowed relations between the elements in ArchiMate in an actual model. These relations are all are equally valid. Note: that there is a ‘core set of relations’ and a ‘core domain’ within ArchiMate may confuse people. These are fully separate concepts.

How the table of allowed relations is calculated is not open to the public, and that way can also not completely be inferred from the text of the standard. What is clear, though, is that there are errors in the table. This post is about an attempt to create a calculation, and an comparison of this alternative to what is actually in the official table. Running ahead of things, 152 relations (including core ones) turned out to be missing from ArchiMate’s official table and 2941 relations in the official table are suspect and  many clearly illegal.

The resulting Excel Spreadsheet and Python program I share under BSD license (so totally free), for all the world to see my feeble first steps at programming in Python :-). And VBA.

[UPDATED Feb 9/10: The Core Set Data in the spreadsheet and the Python program have been updated. Some relations were marked as illegal which shouldn’t and it turned out it was a combination of missing data and an algorithm error. It is important to remember that this is a work in progress and while it does fix several (serious) problems with the official existing table in the standard, it is by no means guaranteed to be perfect, if at all possible given the state of the standard.

NOTE: the distribution now contains a file Archi-relationships.xml which can be used as a stand-in for the existing relationships.xml to get at least many of the missing core set relations, such as Technology Function Accesses Artifact, back].

First things first: I’d like to thank Michael Herman for providing me with an Excel-format of Archi’s (ArchiMate 3.0) relationship table (that saved me a time-consuming job), and for providing me with a piece of VBA that got me started on how to program a VBA macro. I’d like to thank Jean-Baptiste Sarrodie for working with me on the initial attempt in Prolog, which we could not get to work at scale, after which I restarted in Python. The Prolog work laid groundwork for what followed, though.

Derived Relations

In ArchiMate, if you model something in an actual model of the enterprise using those relations in the table, you are allowed to create ‘derived relations’ in your model. E.g. if System Software is Assigned-To Technology Function and Technology Function Realizes Technology Service, then you may deduce that a derived relation exists from System Software to Technology Service. These derivation rules are (mostly) explained in Section 5.6 of the standard. There are two types of derivations-in-a-model that play a role in this story.

The first type of derivation is a derivation of ‘shortcuts’ using dependency and structural relations. A simple example is shown here:

A ‘short cut’ derivation of a chain of structural/dependency relations in an actual model. This type of derivation is also used in the metamodel for calculating all allowed relations from the core set. Note: the Realization relation above is itself already a derived relation. You can use derived relations as input for more derivations. The original ArchiMate research project proved for the then standard that this did not lead to allowing too much.

Here, a Node Realizes a Technology Service which Serves an Application Function. The rule for deriving structural & dependency relations says that if you can create a chain of these relations in the same direction, you may deduce that between both ends of the chain, the weakest relation of the chain holds. In this example, the weakest relation is Serving, which means that (derived, in red) the Node Serves the Application Function.

Secondly, there is a type of derivation that handles derivations of dynamic relations in actual models. An example is shown here:

An example of a derivation of a dynamic relation in an actual model. This type of derivation is also used in the metamodel for calculating all relations from the core set.

Here the rule says that you may move the start- or endpoint of a dynamic relation to the start of a structural relation (in the case of Trigger, only the Assignment) relation. Hence from the Flow from Process A to Process B we can derive a Flow from Process A to the performer of Process B, via the (structural) Assignment relation between the Business Role and Process B. Something likewise can be deduced for the Trigger. As Process B Triggers Process A we can deduce that the performer of Process B Triggers Process A.

The Table in the 3.0 Standard

ArchiMate states that not all derivations that follow from the rules are allowed in an actual model. If the rules lead to a derivation, this derivation is only allowed if the result of the derivation appears in the table of all relations in the appendix.

In addition to the model-type derivation rules above, another derivation is used that can not be used in actual models, but that holds in the metamodel only: derivation by inheritance. The metamodel contains elements that are Specialisations of other elements in the metamodel. E.g. Contract is a Specialisation of Business Object. Note that the standard is confusing, in that it contains two quite different Specialisation relations, one valid in models and one only valid in the metamodel and it calls  them the same: Specialisation. Anyway, taking our example from the metamodel:

Derivation of Relation in the metamodel based on a relation inherited from the parent in a metamodel-Specilisation relation (which differs from a model-Specialisation relation).

One relation from the core set of relations is Business Process Accesses Business Object. Because Contract is a type of Business Object, we may, in the metamodel, conclude that (derived, in red) Business Process may also Access Contract. This is the red relation, above. Section 15.2 explains this for possible extensions of the language, but it is in fact also used in the existing metamodel: Collaborations are meta-model Specialisations, Contract is a meta-model Specialisation and Node has several Specialisations as children, such as Equipment, Device, System Software, etc.

The states in Section 5.6 that the table is populated by using the same derivation rules that are defined for deriving relations in a model on the metamodel. A Note is added that some special handling of Artifact is in place in the metamodel derivations that populate the table, to prevent unwanted derivations, but it isn’t defined exactly. The use of inherited relations in metamodel derivation is mentioned in Section 15.2. But even with that added, the content of the table cannot be entirely deduced from the text and diagrams in the standard in combination with what is said about derivation rules.

Now, our trust in the ArchiMate designers is of course great, but that doesn’t mean the table is perfect.

For instance, the 3.0 table contains Business Object Accesses Grouping, which means, a Business Object can Access a Grouping. Now, the standard says about deriving relations when Grouping is involved, that you’re simply not allowed to derive anything via Grouping and it also says “A grouping element may have any relationship with any element (provided that the element is a possible target element for the relationship).”. Ignoring the fact that it says target where it could possibly be target or source (as most relations have a direction), I can understand Grouping Accesses Business Object, because since a Business Object can be Accessed (passive, as object) so a Grouping must also be able to be Accessed too. But I can’t understand Business Object Accesses Grouping, because a Business Object cannot Access anything, thus, according to the provision in the text of the standard, also not a Grouping.

Because there is no element X for which a relation Business Object Accesses X exists, Business Object Accesses Grouping is not allowed. (Note, the Access in this diagram runs from left to right).

But it is part of the table nonetheless.

Another example of an interesting derivation in the table is Business ProcessRealizesApplication Function. Since ArchiMate 3, the layers of the Core (‘the enterprise itself’) can mingle and the application layer can depend on the business layer and not only the other way around. But you can only go down in layers through the Serving relation, so any derivation should not result in a Realization relation, as Serving is weaker.

No derivation from Business Process to Application Function in ArchiMate 2. Serving is the result in ArchiMate 3.0, but the table (erroneously) contains Realization (in red, above)

The 3.0 table contains several of such errors. Even some ‘foundational’  relations from the core set of relations are missing. E.g. Artifact Realizes Data Object is not there and neither is Technology Function Accesses Artifact. These are not just omissions, they are essential omissions. It’s like the Eiffel Tower missing some essential rivets at the base.

A New Calculation of the Table

It bugs me that “what is and what isn’t in the relationship table” has never been properly documented. Because the ‘lies-to-children‘ answer “the table defines the allowed relations, trust us” is woefully inadequate, and more or less is equivalent to the answer ‘Because!”. We’re mostly past that maturity level in our lives, I would expect. So, I’ve long thought of trying to create my own derivation implementation using what is stated in the standard and adding those few ‘extra’ rules to prevent chaos. Just as a thought experiment and as a contribution to the standard, to make it better. And, in the end I’ve done such a thing (‘better’ is of course debatable), consisting of a spreadsheet (with data and a VBA macro) and a Python program to calculate all the derivations. It was, I must admit, a lot more difficult than expected, mainly because — here and there — the standard is unclear, ambiguous, or incomplete.

First, I’ve created a spreadsheet with all the relations from the ‘core set’ as far as I could gather from the text and diagrams of the standard. I put in a few fixes and indicated several potential others. The sheet is called “MA Core Set”. The cells that are (or could be) adapted are marked in yellow (you can play with this yourself). Some remarks:

The spreadsheet also contains a table of all the relations in the official table (sheet “ArchiMate 3.0 Full Set”. This data has been taken from Archi 4.0.

The VBA macro in the spreadsheet takes such a table and produces columns on the left of the sheet. It also selects the data that you can then copy/paste into a file for the Python program. Note: watch out for end-of-line trouble.

So, taking the “MA Core Set” sheet with the core set of relations, running the VBA macro in that sheet, and using the result in the left-most column as input for the Python program (via copy/paste), then comparing the output with the result of that same VBA macro run in the sheet with the Archi relations (“ArchiMate 3.0 Full Set”), gives us a difference between what is in the table in Appendix B of the standard and what we calculate from our core set and our derivation rules. Note, the format I use is the format of Prolog clauses, because the first (failed) attempt at calculation was done in Prolog. The output can be switched to a CSV format (see below). But as we want to compare with another input set, I generally still use the Prolog format for writing down relation tuples.

A full command run looks like this:

./Derivation.py CoreSet.prolog --outfile Derivation.out --logfile Derivation.log --compare Archi-all.prolog

The result is rather surprising. Thousands of differences in the total set of relations. For one, the standard has 11,761 of them, and I have about 9,561. And those include 155 that the standard has not. The 155 relations that are not in the standard I consider potentially missing in the standard’s table, and the 2364 relations not in my result I consider potentially illegal relations (mostly derivations) in the standard’s table. That number 2364 sounds like a lot, but many are repeats of one single problem or another.

The question of course is: whose result is best? Let’s first explain a bit more of the rules I followed when calculating.

The Rules

Most of the rules I use are of course taken from the text of the standard where it explains these rules. Before I document my adaptations and choices, first have a look at An AchiMate 3 Map (Layers? What Layers! — 1) where you can find a ‘map’ of ArchiMate into:

An ArchiMate 3 Map

I use the domains mentioned here for defining (the limitations of) cross-domain derivation rules.

Then, we say:

And we also limit as follows (this is not per the standard, but these are my ‘hidden rules’):

For all the details, see the Python code and its documentation. It is very complex to make sure the whole business doesn’t explode.

Some details from the Results

Now about the results, and comparing it to what is actually in the table of the standard.

The 152 ‘missing’ relations in the official table:

Some examples from the 2941 ‘wrong’ relations in the official table:

With the over 2300 relations not-derived or blocked by my code (for mostly, I think, understandable reasons) I could go on longer. You can play with it and take your pick. Note: I do not state that 100% of the relations I block or do not derive should not be in the table, but my feeling is, for a large majority this might be a good choice. Some of these are a matter of more arbitrary choices and taste. The difference: my choices are public :-). And given that I release my code under BSD license, it’s easy for people or organisations to adopt it and adapt it. TOG is free to use my code to create a publicly understandable relationship table.

Conclusion

The derivation mechanism in ArchiMate has always been a bit of a naive dream at ‘student paper’ level, and in reality it is not that very important in real models, neither for producing ‘automatic viewpoints’ (which tool can do this?) or for analysis. It’s far more effective in ArchiMate to choose good modelling patterns and then analyse a model regardless of the derivation rules. They are not that important in daily practice.

What  remains is the role of derivations in filling that all-important table, the table that for some ‘defines ArchiMate’ even and that you need to follow if you want to pass your certification.

With regard to that role: ArchiMate 3’s relationship table’s construction is not fully explained and it contains thousands of relations that are questionable, many of them actually provably wrong from ArchiMate’s own definitions. Even some relations from the core set itself are missing. Frankly, the table is a mess, and in my opinion not trustworthy as an arbiter of what is allowed.

Note that — while I am critical of that mess — this story should not be interpreted as being negative on ArchiMate’s practicality. Ultimately ArchiMate remains a very practical instrument. It shares with humans “being practical regardless of not being extremely logical”. Given that enterprises and their behaviour are very human constructs, this conclusion should not surprise us (for a language/grammar that is supposed to be able to describe the reality of enterprises). And that (grammatical) rules and logic are a poor foundation for meaning, we alreade knew from Uncle Ludwig, so the fact that a metamodel full of circularities and simple derivation rules do not simply give only useful results should not surprise us either.

I can’t add the files directly here as WordPress doesn’t allow me to add a zip file, so currently the only way to get them is via the free ‘tool support bundle’ of Mastering ArchiMate, which can be found via the book’s home page. Note, for fixes and changes (within the philosophy of the existing approach), please let me know so I can incorporate it in my release. See the about page for contacting me.

SP: Don’t forget to read Agile & Architecture (on combining architecture governance with agile development methodologies) on the Enterprise Chess site (SP = shameless plug :-))