| Critiquing in the EDC: Survey and Analysis |
|---|
Dipti Mandalia
Payal Prabhu
Statement
At the very forefront of our project, we realized that critiquing in EDC is minimal, and it is only implemented in the reflection space. By not allowing for immediate backtalk from artifacts in the action space, the user (we will use "designer" and "user" interchangeably since the user's of the system are those that design it) loses valuable information that can be offered through the system. Keeping the user's best interest in mind, we believe a mix of active and passive critiquing within the EDC system would be the best implementation.
We read papers on the Janus family of critiquing systems and on the Voice Dialogue Design Environment (VDDE) system to understand:
- The purpose and implementation of the system; what domain it was created for, and if its domain reflects conditions similar to those of the EDC
- The level of critiquing implemented in each system
- How helpful critiquing is in each system and how it can be enhanced to further benefit the design process
We have now realized that our initial confusion was in understanding the critics' rules in the knowledge base. By reading a few initial papers on the Janus family and on VDDE, we realized that the question of "Who makes the rules?" is essential to the design of a critiquing system.
Rationale
The EDC system is currently being worked on at the L3D Center. While extensive work is being done in trying to make the PitaBoard work smoothly, not much attention has been paid to the critiquing aspect of the system. Dipti was hired by the Center to work towards coding in critics in the system. Aside from writing code, she wanted to delve deeper into the theory of critiquing and understand why one form of critiquing should be chosen over the other, what levels of critiquing are the most effective, and finally how effective can critics really be. Since Payal is working with CSLR (another center under the Institute of Cognitive Science), she wanted to work on a project with Dipti that would expose her to some of the work being done at the L3D.
We understood that knowledge is abundant resource however attention is rarely directed towards extracting this knowledge from a system. Critiquing is a mechanism that, if implemented properly, can explore this vast database of information available to the user and the designer and present it at critical points in the design cycle. This project will be a practical understanding of these principles, including those established by the L3D lab. Additionally, by suggesting a critiquing model for the EDC system, members of the L3D will be able to draw on our report to further advance the implementation of critiquing in the system.
Null Hypotheses
- Critiquing that is directly provided from the action space while the language with the artifacts is being formulated is intuitively easy to implement. For example, the action space not allowing the stakeholder to draw a bus route through a house would mean changing some criteria within the EDC system to halt such breakdowns.
- Critiquing in situations where the ramifications of the design are ill formed in nature and coming to a consensus within the group is difficult, are inherently difficult to implement. For example, whether the system suggests a bus stop near a shopping mall would require some discussion and would not always be acceptable by all stakeholders.
Theory
Critiquing is a dialog where the interjection of a reasoned opinion about a product or action triggers further reflection on or changes to the artifact being designed. It plays a central role in the design process by enabling the communication of reasoned opinion. A Critiquing System is a decision support system that allows the user to make the decision first; the system then gives its advice when the user requests it or when the user's decision is out of the system's permissible range. Basically, a critiquing system is anything that takes a problem description and a proposed solution as input and produces a commentary on that solution as output.
Systems like the EDC need critiquing because the problem is ill-defined and thus problem formulation is incomplete at any given point. For example, the Hydra system (Janus) is developed for good kitchen design but the definition of "good" is ill-formed. Additionally, critics help in assessing the solution presented by the designer. Another big factor is the involvement of a community of people consisting of bus-route designers, system designers, commuters, and the general population of the town. These calls for a multitude of perspectives that can often be conflicting yet have to be integrated into the design process.
One of the major design questions is whether to use active or passive critics. In order to decide between the two, we need to first understand what each one entails. Active critics are those that interject the designer as soon as a conflict between the proposed solution and the rule base results. This implies that such a rule base already exists in the system and is ready to be compared against at all times in the process. Active critiquing has to be implemented extremely carefully for fear of bombarding the designer with too much information and thus restricting social creativity. On the other hand, passive critics "lay low" while the designer explores through solutions and then critiques the proposed final solution (at any stage of completion) only when the designer explicitly requests critiquing. For this reason, it is often also called "after-task batch critiquing".
In general, the knowledge base for any system is divided into two or more parts based on the severity of conflicts that result from the proposed solution. Depending on what form of critiquing the designer selects one or more parts of this knowledge base are presented for evaluation. For example, if the designer chooses active critiquing s/he risks being criticized too often, yet could access more information from the knowledge base. The level of the designer’s experience with such a system is relied upon for selection of parts of the knowledge base, as well as for selection of active or passive critiquing.
Contribution
We propose a model that allows for both active and passive critiquing which are offered as requested by the designer. Active critiquing can be very responsive (watchdog-like) however it can become too literal without involving any deeper thought into the design process. Passive critiquing can be much smarter but in domains like EDC, smartness at the end of the whole design process does not contribute much to the actual design of the system. This is why we think a balance is needed between these two kinds of critics and we suggest such balance for critics in the EDC.
On reading, Sumner et. al. we came to the conclusion that active critiquing is an acceptable solution to all users and designers of a system regardless of the level of experience they have had with the system. And, on analyzing the EDC domain, we understood that the active critiquing should be implemented in the action space. This solution makes logical sense since the critic would be involved with the design process while the designer's focus is on the PitaBoard and would not require the designer to shift between the action space and the reflection space.
In the model that we propose, the knowledge base is divided into two parts. The first part contains rules that cannot be violated under any circumstance. For example, rules that deal with changing major bus-stops in an established route are not to be violated ever. If such a conflict arises during the course of the design process, the designer is immediately notified of the dispute and is allowed to continue only when the problem is rectified. This enables the system to enforce the established standards of "bus route design". The second part of the knowledge base contains rules that can be violated though only temporarily. An example of this sort of rule would be placing a new bus-stop outside of the preferable walk-perimeter of a commuter. Although ideally no commuter should be inconvenienced, equilibrium for the majority of commuters often comes at the expense of a handful of them. Again, violations of these type of rules are prompted to the user when requested and a user is allowed to ignore the criticism and carry-on with the design process.
Intrusiveness is controlled completely by the designer in this model. At the very start of a session, the EDC system gives the user the option to select both Part 1 and Part 2 to be used in active critiquing, or only use Part 1 immediately and Part 2 on demand. Depending on his/her own prior experience with the system (and hence his/her comfort with knowing the rules of design and knowing the frequency of criticism), the designer is able to regulate how often the artifacts are involved directly in designing the bus-route. For example, an expert user (like a representative from RTD) will more often than not be criticized only from Part 2 because rules in Part 1 of the knowledge base are "common knowledge" to them given their vast experience with designing such bus-routes. This is unlike the situation in which a novice user (like a regular commuter using the bus system) who would (intuitively) be unaware of the established rules of bus-route design and falter often in proposing solutions.
Concerns with Proposed Solution
As pointed out at the beginning of this report, the problem of "Who makes the rules?" still exists. We assume that rules in Part 1 can be provided by RTD designers who have had much experience in this area. However, given a large democratic community, some of these "established" rules can be challenged by a large-enough group forcing the designers to make concessions. Again, to what level should these concessions be allowed? How large of a group must contest these rules to enable a change in the knowledge base? Additionally, rules in Part 2 can be made from individual concerns expressed by those that attend such meetings.
Technical problems with this solution also exist. First, the current EDC system implementation is too modular to implement unified rule base as we suggest; an attempt should be made to make it as declarative as possible. Second. labeling roads as primary, secondary and tertiary is hard to implement (based on personal judgment). Third, the real-time nature of the EDC system allows choices to be far from binary (it allows a user to move the road to suit the bus-stop; to move a house/school/mall to suit the route; etc.). And finally, the rules are not mutually exclusive for this domain. Changing one rule may affect another in the rule base.
Future Work
What we propose is a model for a critic in the EDC system. Given the time constraints of a semester, implementing this model is not feasible for us. It would be interesting to see if this solution works as well as we think it would and then to evaluate the breakdown points as and when they arise.
We think implementing this critic would be an interesing summer project for an intern in the Computer Science Dept.
If this solution becomes buggy, further theoretical investigation can be done to find other ways to make critiquing an effective solution to the EDC domain.
Reflections
On presenting our ideas to the class on Monday April 29th, our solution for critiquing was well challenged by members of the L3D and the students in class. One of the concerns brought up by a current EDC staff member was the rigidity involved in critiquing with rules from Part 1 of the knowledge base. Specifically, "if a church was to be torn down in place of a new road, how would you bypass a rule in Part 1 that said no buildings should be removed to accomodate the new bus route?" After ruminating on this question for a while, we decided on a revised solution.
In case a rule in Part 1 is violated, the designer is given the choice to either ignore or accept (and make changes to the system) before proceeding with the design process. If the designer chooses to ignore the rule, the system will require additional input. For example, the designer would have to shift focus from the action space to the reflection space and input a valid reason for wanting to bypass the rule in this particular case. Only on completing this part of the process, would the designer be allowed to continue the process in the action space. In effect, when we suggest that the rule base be divided into two parts, the demarcating criterion should be the freedom given to the user to assess the relevance of the critic. Also we think this "shift of focus" would be a good idea because:
- by forcing the designer to shift focus temporarily, we could enforce better reflection on the part of the designer so that s/he can be really sure that bypassing the rule is a good strategy for the current case
- there will be a record of any such rule bypassing so that reflection at later stages will be facilitated from the knowledge base (when you wonder "why" you did not accept the system's criticism of your solution)
- (as Hal suggested) by enforcing the rule-bypassing mechanism per-rule per-criticism, the designer would be able to re-incorporate the rule from the knowledge base if it were critiqued another time (rather than switching off the rule once and for all)
- Also this would lead to the use of the rerlection space and incorporate the most important component of the EDC; the reflection space into the crtiquing mechanism. The reflection space is inherently more powerful at helping the designer reflect on the partial design
Also, Hal suggested the incorporation of a "facilitator" in the design process who would be a go-between for historical system designers and "current" systems designers (users of the current bus route). We think this would be a great idea since it would bring in the "human" component to critiquing in the EDC system. The facilitator can also be the one who will modify the severity of the rules in both the rule bases, there by tailoring them to the current design scenario. This will help us tackle the problem of the relevance of the critics in all design scenarios to some extent (as we know that not all rules work all the time, the facilitator can modify the rules a little as and when needed). Not only as someone who would be able to mediate decisions between both types of designers but also as someone who would be a "record" of activity on the action space, facilitators could prove immensely beneficial in the process. Their record-keeping task could later be alleviated by more sophisticated sytems that have audio/video support for recording each session and integrating it smoothly into the critiquing mechanism so that current designers (in the session) would be able to avail of knowledge from previous designers (in previous sessions). We understand this record-keeping task is already being tackled by Hal and his assistant Jack. However care should be taken that the critiquing mechanism of the EDC works efficiently in the absense of the facilitator too. So we think that human intervention in the critiquing process is one more aspect where the EDC needs a balance.
References
- Antibody Identification Assistant (AIDA), An Example of a Cooperative Computer Support System (Guerlain, S. et.al.)
- Crack: A Critiquing Approach to Cooperative Kitchen Design (Fischer. G., Morch, A.I.)
- Design Critiquing Systems (Robbins, J.E.)
- Embedding Critics in Design Environments (Fischer, et.al.)
- Intelligent Critic System for Architectural Design (Chun, H.W., Lai, E.M.)
- Making Argumentation Serve Design (Fischer, et.al.)
- Perspective Based Critiquing: Helping Designers Cope with Conflicts Among Design Intentions (Nakakoji, K., Sumner, T.)
- Real-time Critiquing of Integrated Diagnosis/Therapy Plans (Gertner, A.)
- Role of Critiquing in Cooperative Problem Solving (Fischer, et.al.)
- Transcending the Individual Human Mind: Creating Shared Understanding Through Collaborative Design (Fischer et.,al.)
- Supporting Evaluation in Design (Sumner, T., Bonnardel, N.)
- MS Thesis, Univ. of Colorado, Boulder (Harstad, B.)
          
|