Qualitative Comparison of Agent-oriented Software Engineering Processes


Following Tran et al. (2005), we evaluate PosoMAS for different process criteria, the coverage of different steps and of multi-agent system concepts. A comparison with the processes O-MasE, Prometheus, and ASPECS is provided as well. The data for Prometheus is taken from Tran et al. (2005) and Padgham and Winikoff (2005), the evaluation for O-MasE is based mainly on DeLoach and Garcia-Ojeda (2010), and the one of ASPECS mainly on Cossentino et al. (2010).

Process-related, Model-related, and Supportive Feature Criteria

The evaluation of the different process criteria for PosoMAS and the comparison with the reference processes is depicted in Table 1. Details of the evaluation for PosoMAS and ASPECS are given in Table 2. It is important to note that the tables only capture if a process has explicit supportive content for a certain criterion. It is, e.g., very much possible to build proactive multi-agent systems with PosoMAS even though the process does not include specific support for them.

Table 1: Comparison of PosoMAS with O-MasE, Prometheus, and ASPECS based on the evaluation criteria
detailed in Evaluation and Validation Criteria and in Tran et al. (2005). Data for Prometheus from Tran et al. (2005), data for O-MaSE based on DeLoach and Garcia-Ojeda (2010), data for ASPECS based on Cossentino et al. (2010), data for Gaia from Tran and Low (2005), Tran et al. (2005), and Cernuzzi et al. (2014), and data for Tropos from Tran and Low (2005) and Morandini et al., and data for INGENIAS from González-Moreno et al. (2014) and Gómez-Rodríguez et al. (2014).
Select visible processes:
Evaluation Criteria PosoMAS O-MaSE Prometheus ASPECS Gaia Tropos INGENIAS
Process-Related Criteria
Development life-cycleIs the life-cycle formally defined? Which life-cycle is adopted (e.g., Waterfall)? Iterative-incremental risk-value life cycle Depends on base process Iterative across all phases Iterative-incremental life cycle Iterative within each phase but sequential between phases Iterative, non-incremental life cycle Iterative-incremental life cycle
Coverage of the life-cycleWhat disciplines are covered by the methodology (e.g., analysis, design, implementation). Conceptualisation, Analysis, Design, (Test, Deployment, Management) Analysis, Design Analysis, Design Analysis, Design, Test, Deployment Analysis, Design Requirements, Analysis, Design, Implementation, Testing Analysis, Design, Implementation, Deployment
Development perspectivesDoes the methodology support both bottom-up and top-down development or does it follow a hybrid approach? Hybrid Top-Down Bottom-UpDesign in Prometheus is driven from the agent perspective. While there is a "System Overview" activity in the Architectural Design phase, it is executed last and deals with data representation, percept processing, and action management instead of high-level system design. Top-Down Top-Down Top-Down Top-Down
Application DomainCan the methodology be applied to arbitrary domains or is it suited for specific domains? Any (emphasis on open systems) Any Any Any Any Any (emphasis on socio-technical systems) Any
Size of MASWhich system size does the methodology support? Not Specified Not Specified Not Specified Not Specified < 100 agents Not Specified Not Specified
Agent paradigmIs the methodology suited for the development of agents of a specific paradigm (e.g., BDI agents) or is it independent of the reasoning mechanism and meta-model used? Heterogeneous Heterogeneous BDI Holonic Heterogeneous BDI BDI
Support for model validationAre there provisions for the validation and verification of the models and specifications developed in the methodology? YesPosoMAS allows the validation of domain and design models w.r.t. the requirements if formal specification of requirements and the practice "Model-driven Observer Synthesis" is used. Consistency ChecksO-MaSE claims to have verification abilities, but these are limited to consistency checks between models and do not amount to formal verification of agent behaviours or interaction in the classical sense. Consistency and Completeness Checks No No Yes Consistency and Completeness Checks through transformations and reuse
RefinabilityDoes the process define how a model is gradually refined and augmented in the course of the process? Yes Yes Yes Yes Yes Yes Yes
Approach towards MAS developmentDoes the methodology apply a specific MAS development approach (e.g., based on methods form knowledge engineering, object-oriented, role-based, or "non-role-based", i.e., relying on other means such as use cases, workflow models, or interactions)? Object-Oriented, Non-Role-Based Object-Oriented, Role-Based, Goal-Oriented Object-Oriented, Non-Role-Based Role-Based, Knowledge Engineering Object-Oriented, Role-Based Goal-Oriented (i* Modelling Framework), Non-Role-Based Goal-Oriented, Role-Based
Meta-model basedDoes the process prescribe a meta-model as the basis for architectural models of the system? No Yes No Yes Yes Yes Yes
Model-Related Criteria
Syntax and SemanticsAre the syntactical elements of the models and their semantics clearly defined? MediumPosoMAS does not prescribe a formalism to express design diagrams but favors UML for which it also suggests a profile containing a number of helpful stereotypes. However, meta-model driven processes have a stronger semantic backing than PosoMAS. In other areas, e.g., requirements engineering, PosoMAS in turn favors well-described formalisms while others are more lenient. High High High High High High
Model transformationsIs the transformation of models as part of a model-driven engineering approach supported by providing guidelines for the transformations? Yes Yes Yes Yes Yes Yes Yes
ConsistencyDo provisions exist that guarantee internal consistency within a model and between models? Is it possible to check consistency between levels of abstractions and between different representations of the same aspect in different models? Yes Yes Yes Yes Yes Yes Yes
ModularityCan agents be structured in a modular way? Yes Yes Yes Yes Yes Yes Yes
AbstractionIs it possible to model the system and the agents at different levels of detail and abstraction? Yes Yes Yes Yes Yes Yes Yes
AutonomyDoes the methodology support modelling the autonomous features of an agent? Yes Yes Yes Yes Yes Yes Yes
AdaptabilityAre features of adaptability such as learning supported by the models? No Yes No Yes No Yes Yes
Cooperative behaviourDoes the methodology support modelling the cooperative behaviour of the agents? Yes Yes Yes Yes Yes Yes Yes
Inferential capabilityIs it possible to model automatic agent inference, i.e., the capability of an agent to derive concrete actions from abstract commands? No Yes Yes No No No No
Communication abilityDoes the methodology support modelling communication and knowledge exchange between agents? Yes Yes Yes Yes No Yes Yes
PersonalityDoes the methodology support modelling an agent's "ability to manifest attributes of a `believable' human character"? No No No No No No No
ReactivityIs there support for the modelling of reactive agents that act as a response to sensed stimuli? Yes Yes Yes Yes Yes Yes No
ProactivityDoes the methodology support modelling proactive agent behaviour, i.e., the ability of the agents to self-initiate a deliberate act to achieve a certain goal? No Yes Yes Yes Yes Yes Yes
Temporal continuityCan the models support and represent temporal continuity of agents (i.e., persistence of identity and state over long periods of time)? No No No No No No No
ConcurrencyDo the models prescribed by the methodology allow capturing concurrency of and synchronisation between processes? No Yes No No No No No
Model Reuse Does the methodology provide, or make it possible to use, a library of reusable models? No Yes Yes Yes No No Yes
Supportive Feature Criteria
Software and methodological supportAre there tools and libraries of reusable components that support the methodology? Yes Yes Yes Yes No No Yes
Open systems and scalabilityDoes the methodology provide support for open systems that allow agents and resources to dynamically enter and leave the system? Yes No No Yes Yes No No
Dynamic structureIs there support for self-organisation processes, i.e., the dynamic reconfiguration of the system structure? Yes Yes No Yes No No
Performance and robustnessAre techniques for dealing with exceptions, capturing error states and recovering from failures as well as performance monitoring provided? Yes No Yes Yes No No No
Support for conventional objectsCan "ordinary objects" be integrated into the design and are interfaces to such objects captured? Yes No Yes Yes No No No
Support for mobile agentsAre there possibilities to integrate mobile agents in the developed MAS? No No No No No No No
Support for self-interested agentsIs there support for agents that do not adhere to the benevolence assumption and thus do not necessarily contribute to the overall system goal? Yes No No Yes No No Yes
Support for ontologiesCan ontologies be integrated in the design of the MAS? No No No Yes No No No

While O-MasE does not explicitly prescribe the BDI agent paradigm, the elements of the used meta-model strongly suggest that the method engineers who created it had current BDI implementations in mind. The OMACS meta-model used defines elements for goals and plans. Desires and Intentions are handled exactly with these concepts in procedural reasoning engines such as Jadex. Indeed, goals exist on two levels in O-MaSE: on the organisational level they direct the creation of an organisational structure to achieve the goals, on the agent level, they define individual goals. The latter goals are achieved by plans (as in BDI), the former goals by assigning roles.

The methodology has a philosophy similar to PosoMAS: method content is defined independently of a process life-cycle. Therefore, some of the aspects—e.g., the development life-cycle—depend on the base methodology that is combined with O-MaSE's method content. PosoMAS, however, is also a process. Therefore, the "development life-cycle" criterion in Table 1 is evaluated differently. Unfortunately, the O-MaSE method fragments are not available in a repository, although the content is distributed with the download of agentTool, the program also used for the creation of the required models.

Prometheus is the oldest process in the list, but is still one of the best described ones and is still frequently cited. While it does not insist on a meta-model, it prescribes the use of BDI agents. Therefore, the methodology concentrates on the definition of goals, plans, beliefs, and events. The development is mainly from the bottom up since it is very focused on individual agent behaviours and their constitutive elements. One of the bigger differences to the other processes is the fact that adaptability is not explicitly taken into account.

The relatively recent process ASPECS is probably the one most similar to PosoMAS. The authors are veterans of agent-oriented software engineering—see, e.g., PASSIM (Cossentino et al., 2008)—and in many regards, ASPECS is the culmination of their collectively learned lessons. The process is also described on a specialised website, although the content is very similar to the one from the paper. ASPECS too employs recent AOSE techniques and is aimed at complex systems. However, there are important differences to PosoMAS, especially with regard to the level of abstraction. The main concept ASPECS deals with are holons. If a system can not be described in holonic terms, the process is not applicable. In addition, ontologies are a central focus of the design of the system. They are used to capture the concepts of the problem domain and used extensively in later stages of the design as well as in the communication between the agents—the traditional use of ontologies in MAS. Cossentino et al. (2010) argue that using ontologies enhances the design and allows using AI technologies, possibly semantic-web approaches.

ASPECS also prescribes meta-models for the "agency domain", the "problem domain", and the "solution domain". The first defines concepts such as tasks, groups, roles, and communication, while the second deals with organisations, interactions, and ontologies. The solution domain includes concepts used in the implementation and can contain platform-specific choices. The three meta-models are the basis of the model-driven approach followed by ASPECS in which models are successively refined and transformed from one domain to the next. This ensures consistency of the models but requires the development of transformation rules. PosoMAS and ASPECS are compared according to the criteria from Table 1 in more detail in Table 2. Please note that Cossentino et al. (2010) includes a similar comparison with other processes based on a criteria set "inspired" by Tran et al. (2005).

Detailed comparison of PosoMAS and ASPECS according to the evaluation criteria described in Evaluation and Validation Criteria.
Evaluation Criteria PosoMAS ASPECS INGENIAS
Process-Related Criteria
Development life-cycle Based on OpenUP's risk-value life cycle. Iterative across and within all phases. Based on UDP's risk-value life cycle.
Coverage of the life-cycle Explicit agent-focused support of conceptualisation with goal-based requirements analysis, design with a number of practices, and partially implementation, e.g., with model-driven observer synthesis. Support for implementation and deployment through OpenUP. Claims to support all phases but is focused on analysis and design, specifically agent and holon design. Covers implementation with rudimentary activities. Focused on analysis, design, and deployment with additional method content to cover parts of the implementation with automatic code generation for JADE.
Development perspectives Analysis is performed in a top-down fashion while the design supports both perspectives. Top-down design is used in agent architecture design and system design, bottom-up approaches in the specification of the self-organisation algorithm. Analysis starts with the definition of the environment and the overall system architecture. From that, tasks and goals are defined and successively refined until the level of the individual agent models is reached.
Application domain In principle any domain, although adaptivity and heterogeneity are a focus. In principle any domain, although holonic approach could limit this. In principle any domain, although competitive and open systems are not directly supported.
Size of MAS Aimed at large-scale heterogeneous projects with many agents not under the control of the original developers. Aimed at large-scale projects that profit from holonic perspective. Not specified.
Agent paradigm All method content is formulated in a paradigm-agnostic way. The agency meta-model does not prescribe a concrete agent paradigm. The focus on goals and the code generation are aimed at the BDI paradigm and its concrete instantiation in JADE.
Support for verification The use of constraints within the process allows a limited form of formal analysis. Although Cossentino et al. (2010) states that support for formal analysis is available, no information is provided either in the paper or on the website. Formal analysis is not part of the process.
Refinability Advocated by the practices Evolutionary Agent Architecture and OpenUP's Evolutionary Design. Stepwise refinement of the models is an important part of the proces. Embedded in the process in several activities for the different types of models.
Approach towards MAS development While PosoMAS does not explicitly prescribe the approach, its practices are formulated in a way that strongly favours an object-oriented approach. The definition of roles and of agent organisations is promoted. In addition, ontologies play an important role, so knowledge engineering applies as well. The development of the MAS is based on tasks and goals for the entire system that are refined to agents and the roles they play.
Meta-model based No. PosoMAS does not prescribe any meta-models. ASPECS defines meta-models for the "agency domain", the "problem domain", and the "solution domain". The INGENIAS meta-models cover agents, organisations, tasks and goals, environment, and interactions.
Model-Related Criteria
Syntax and Semantics Syntax and semantics of the models are inherited from UML which is used as a standard modeling language. While this is a different level than using meta-models, it still ensures that the models are based on clearly defined concepts. Enforced by the meta-models. Enforced by the meta-models and by the model transformations.
Model transformations Yes, e.g., in Model-driven Observer Synthesis. Yes, the process explicitly promotes using model-driven engineering. Yes, the process includes activities to define the transformation templates and perform the transformations itself.
Consistency Ensured by evolutionary refinement, model transformations, and corresponding guidelines. Ensured by model transformations and strong adherence to meta-models. Ensured by model transformations and strong adherence to meta-models.
Modularity Modular agents can be composed from individual components or with other object-oriented techniques. Modularity based on roles, ontologies, and object-oriented techniques. Modularity based on tasks, goals, and roles.
Abstraction Yes, models are possible on different levels of abstraction.
Autonomy Yes, by the definition of observation and control models. Yes, based on individual and collective agent goals. Yes, based on agent and system goals, tasks, and roles.
Adaptability Yes, by the observation and control models, the self-organisation algorithm, and the trust-based interactions. Yes, by adapting the holonic structure and with autonomous agent decisions. Yes, by autonomous agent decisions.
Cooperative behaviour Yes, especially with regard to incorporating untrustworthy agents. Yes, within holons and between holons. Yes, by sharing common goals.
Inferential capability Not included. Not included. Not included.
Communication ability Yes, based on FIPA protocols and Trust-based Interaction Design. Yes, based on FIPA protocols and supported by the ontology. Yes, based on the interactions meta-model.
Personality Not included. Not included. Not included.
Reactivity Yes, by the definition of observation and control models. Yes, with events. Not included.
Proactivity No, there is no specialised method content to account for proactivity. Yes, based on individual and collective agent goals. Yes, based on agent and system goals.
Temporal continuity Not included. Not included. Not included.
Concurrency Not included in the processes, but can be modelled with standard UML constructs such as parallel regions in state charts. Not included.
Model Reuse Yes, models are reused for different purposes within both processes.
Supportive Feature Criteria
Software and methodological support Yes, standard CASE tools can be used. Guidances include many reusable assets. Use of the INGENIAS Development Kit is enforced.
Open systems and scalability Yes, through Trust-based Interaction Design, Agent Organisation Design, and support material. Yes, through the use of holons and ontological support. Not explicitly supported.
Dynamic structure Yes, through Agent Organisation Design. Yes, through the use of holons. Not included.
Performance and robustness Yes, through Trust-based Interaction Design, Agent Organisation Design, and the observation and control models. Yes, scalability through the use of holons. Not explicitly supported.
Support for conventional objects Yes, with the use of standard UML. No.
Support for mobile agents Not included. Not included. Not included.
Support for self-interested agents Yes, through Trust-based Interaction Design Yes, through individual goals and autonomous decisions about participation in holons. Yes, through individual goals.
Support for ontologies Not included. Yes, extensively. Not included.

Process Steps

The technique-related criteria shown in Evaluation and Validation Criteria are the basis for a metric to evaluate how well the steps are covered by the different processes. The list below explains the different levels of coverage used in Table 3.

None of the processes used in this comparison covers all the process steps. As described in Evaluation and Validation Criteria, this is not strictly necessary although the steps capture many important activities within an AOSE process.

Table 3: Coverage of the 19 required process steps introduced in Tran and Low (2005), according to the metric shown above. Data for Prometheus from Tran et al. (2005), data for O-MaSE based on DeLoach and Garcia-Ojeda (2010), data for ASPECS based on Cossentino et al. (2010), data for Gaia from Tran and Low (2005), Tran et al. (2005), and Cernuzzi et al. (2014), data for Tropos from Tran and Low (2005) and Morandini et al., and data for INGENIAS from González-Moreno et al. (2014) and Gómez-Rodríguez et al. (2014).
Select visible processes:
Steps PosoMAS O-MaSE Prometheus ASPECS Gaia Tropos INGENIAS
Problem Domain Analysis
1. Identify system goals Full Full None Full None Full Full
2. Identify system tasks/behaviour Full Full Full Full Full Full Full
3. Specify use case scenarios None Full Full Full None None Full
4. Identify roles None Full None Full Full None Full
5. Identify agent classes Full Full Full Full Full Full Full
6. Model domain conceptualisation Full Partial Partial Partial1 None Exemplified Limited2
Agent Interaction Design
7. Specify acquaintances between agent classes Limited Full Full Full Full Full Full
8. Define interaction protocols Full Full Full Full Full Full Full
9. Define content of exchanged messages Full Full Included Full None Full Full
Agent Internal Design
10. Specify agent architecture Full Full None Full None None Limited3
11. Define agent mental attitudes (e.g., goals, beliefs, plans, commitments, ...) Partial None Full Partial None Full Full
12. Define agent behavioural interface (e.g., capabilities, services, contracts, ...) Included None Full Partial Full None Partial
System/Environment Design
13. Specify system architecture (i.e., overview of all components and their connections) Described None None Partial None None Full
14. Specify organisational structure/control regime/inter-agent social relationships Full Included None Full Full Included Partial
15. Model MAS environment (e.g., resources, facilities, characteristics) Described None Described Partial Partial Partial Full
16. Specify agent-environment interaction mechanism Full None Full Partial None None Full
17. Specify agent inheritance and aggregation Included None None Included Full None None
Deployment
18. Instantiate agent classes None Full None Full Included None Full
19. Specify agent instances deployment Included Full None Full None Included Full

1: ASPECS describes the domain and conceptualises some of its aspects in the problem domain ontology but does not make them available as software engineering concepts.

2: INGENIAS offers limited domain modelling in the use cases ("Problem definition" artefact) and in obtaining the environment model but does not offer explicit notation or activities for the purpose.

3: Since INGENIAS prescribes the multi-agent framework that must be used and is limited to the BDI paradigm, variations in the agent architecture are severely limited

Coverage of Multi-Agent Concepts

Apart from the validation criteria and the process steps, Tran et al. (2005) also provide a list of multi-agent systems concepts and how well the respective processes cover these concepts. The coverage of concepts provided by PosoMAS and the comparison to O-MasE and Prometheus are shown in Table 4.

Table 4: Comparison of the support for agent concepts between PosoMAS, MasE, and Prometheus as suggested by Tran et al. (2005). Data for Prometheus from Tran et al. (2005), data for O-MaSE based on DeLoach and Garcia-Ojeda (2010), data for ASPECS based on Cossentino et al. (2010), data for Gaia from Tran and Low (2005), Tran et al. (2005), and Cernuzzi et al. (2014), data for Tropos from Tran and Low (2005) and Morandini et al., and data for INGENIAS from González-Moreno et al. (2014) and Gómez-Rodríguez et al. (2014).
Select visible processes:
Concepts PosoMAS O-MaSE Prometheus ASPECS Gaia Tropos INGENIAS
Problem Domain
System goals
System roles
System functionality/Tasks
Task responsibilities/Procedures
Design requirements
Use case/Scenarios
Agent Properties
Agent classes
Agent instances (including cardinality)
Agent's roles
Agent's functionality
Agent's knowledge/beliefs
Agent's plans
Agent's goals
Agent's capabilities
Agent mobility
Agent Interaction
Interaction pathways
Exchanged messages
Interaction protocols
Interaction constraints
Conflict resolution mechanisms
Contracts/commitments
Ontology
Agent Relationships
Inheritance
Aggregation
Association
System/Environment
Co-existing non-agent entities
Infrastructure/environment facilities
Organisational Structure
Agent-environment interaction
Environment characteristics
Deployment
Agent architecture
System architecture
Location of agent instances

Bibliography

Copyright

This material is made available under the Creative Commons—Attribution-ShareAlike License v3.0.

© Copyright 2013, 2014 by Institute for Software & Systems Engineering, University of Augsburg
Contributors
Contact: Jan-Philipp Steghöfer