Introduction

This document describes a simple model of XSDL schema composition in Alloy . One goal is to help clarify the design of composition; another is to generate test cases for testing the behavior of existing processors. If we know what kinds of cases are treated consistently by existing XSDL software, we have a better chance of ensuring that the description of schema composition in XSDL 1.1 does not unnecessarily cause interoperability problems between 1.0 and 1.1 processors. Also, empirical information on existing processor behavior can suggest places where XSDL 1.0 is underspecified, contradictory, unclear, or too subtle.

The document first discusses aspects of schema composition we may want to model, then addresses the problem of building test cases with interpretable results. Then a simple model of schema composition is presented in Alloy notation, and some sample instances of the model are presented. Section describes the form of test cases to be generated from instances of the model; section describes a small number of tests hand-made on the basis of Alloy instances (and in some cases from user reports) and the results of running those tests on existing processors. Section describes code that generates test cases automatically from Alloy model instances; and section summarizes the tests generated in this way and the results obtained from running them.

In this version of this document, familiarity with both XSDL and Alloy is assumed. If they have sufficient tolerance for partially understood technical detail, readers lacking some of that knowledge may nevertheless find it comprehensible in parts.

Sources of variation

To generate test cases with reasonably good coverage of schema composition, it would be convenient to be able to model the points on which the behavior of processors may or must vary. These include: processor policy for component acquisition: are schemaLocation hints followed? In principle, the policy could be anything; for testing, we probably need to specify a choice of: follow no schemaLocation hints; read only the schema documents specified at invocation time (this involves using a proxy server that ensures that the required attempts to dereference the schema documents will fail) follow schemaLocation hints only when required, i.e. only on xs:include and xs:redefine elements; ignore schemaLocation hints on xs:import and in xsi:schemaLocation and xsi:noNamespaceSchemaLocation attributes in the instance follow the first schemaLocation hint for each namespace not covered by a schema document named at invocation time follow all schemaLocation hints (even multiples) Note that the existing XML Schema test suite appears to be built on the assumption that the choice of processor policy for component acquisition is unproblematic. The test suite catalog does allow several schema documents to be specified for a test, using the schemaTest element and its schemaDocument children. It is not clear from the documentation, however, whether processors running the test are expected to read those schema documents and no others, or to read all of those schema documents and optionally any others they may point at. starting documents: which schema documents are specified at invocation time? document availability: for each set of documents, some subset may be unavailable schemaLocation hints in instance: for each namespace, does the instance point to a starting schema document for that namespace? Does it do so once, more than once, or not at all? In what order do the schemaLocation hints occur? order of lists: when several schema documents are specified at invocation time, or in the same schemaLocation attribute, what order are they listed in? Does it matter?It clearly does matter sometimes. Consider an instance beginning ]]> At least two processors build a schema and validate the instance without complaint, but if a second namespace / schema document pair is added to the schemaLocation attribute, they fail to find the declaration for the ns1:wrapper element. ]]> If the order of the schema location pairs is changed, the behavior of the processors changes. schemaLocation information in the schema documents: for each pair of schema documents A and B, does A include B? B include A? import? redefine? N.B. I'll use the verb call to cover all three of these relations: A calls B if and only if A includes B or A redefines B or A imports B. single call vs. multiple call: if A includes B, does it do so once? twice? how many times? Ditto for import and redefine.

It's not clear what interpretation the XSDL spec supplies for schema document A containing two xs:redefine elements each pointing to document B and redefining the same or different components. But I have not noticed any rules which explicitly forbid it. The example serves a useful purpose either way: either we can demonstrate that the spec forbids it, or we can demonstrate that the spec allows it and assigns a particular interpretation to it, or we cannot demonstrate either of those propositions, which is itself a useful, although not a particularly happy, result.

namespace matching: when A calls B, do their namespaces match or differ? The constraints in XSDL are straightforward enough to check that there doesn't seem much point in trying to exercise all possible variations here; it's probably enough to have some simple examples of violations of these rules, and then to generate other test cases from schema documents which obey these rules: if A includes or redefines B, then their target namespaces match, or B lacks a target namespace if A imports B, then their target namespaces differ I don't see rules forbidding cycles on any kind of composition self-inclusion or self-redefinition (N.B. self-import fails the namespace-cleanliness rules) multiple calls to the same schema document of different types multiple calls of the same type to the same schema document behavior with respect to missing components; the possible kinds of missing component appear to be: See , Appendix A, for slightly more detail. attribute {type definition} element {type definition} element {substitution group affiliation} type definition {base type definition} type definition {attribute uses} type definition {content type} (simple type may be unavailable) type definition content model may refer to unavailable elements attribute group may refer to unavailable attributes named model group may refer to unavailable elements element identity constraints may be unavailable simple type definition {member type definition} simple type definition {item type definition} It might simplify life if we can successfully test schema composition without getting into missing components. lazy assembly vs. eager assembly: does the processor gather all schema documents together first, then build the schema, then validate? Or does it begin validation with a partial schema, to which components are added when needed? In principle, the XSDL spec has been written to make lazy assembly possible, but we are not supposed to be able to tell the difference. And indeed one rationale for the hand-waving in XSDL's description of component acquisition is that leaving processors such freedom allows any odd results of lazy assembly to be understood instead as quirky choices during component acquisition. It's not clear whether test cases should, or can, actually expose the reality of the processor's behavior in this area, but it may be possible to construct test cases which present different challenges for lazy and eager assemblers.

A simple model of schema composition

Trying to generate test cases to cover all possible combinations of the variables identified above may be very time-consuming and produce more test cases than we can conveniently handle. If we assume three schema documents some subset (zero to three) of documents specified as starting documents some subset actually available some subset of schema documents pointed to from the instance, in either or both of two locations some network of call relations (include, redefine, import) among schema documents if one schema document calls another, it may do so either once or twiceStrictly speaking, it might do so any number of times, but it seems unlikely that processors will handle two such calls but not three, or vice versa. So we limit the number to two, for simplicity in the calculation we are performing. then there are 8 × 8 × 64 × 7,625,597,484,987, I.e. (2 ** 3) sets of starting documents, × (2 ** 3) sets of documents actually available, × (4 ** 3) distinct patterns of schemaLocation hints from the instance, × (27 ** 9) different general labeled graphs of call relations. or about 3.1E16 (3.1 quadrillion) cases to be generated.

Even if these three quadrillion test cases could be generated with acceptable speed, it might not be easy to find people willing to run them all for various processors. It may be a good idea, therefore, to try to divide the space a little and make some simplifying assumptions in exchange for a more tractable set of test cases.

We are also not necessarily required to generate all possible test cases; a systematic or random selection might suffice to point to important unanswered questions.

In the spirit, therefore, of starting with a fairly simple model of the problem, we can begin by ignoring some aspects of schema composition and modeling just part of the phenomenon.

We are unlikely to need to reuse this model, but in case we do, and for the sake of simple documentation, we define this model as an Alloy model named schema_composition01. The body of the model consists of some rules about namespaces, some about schema documents, and finally some miscellaneous predicates. module schema_composition01 // First simple model of schema composition. Think of this // as a sanity check. // This model oversimplifies by not considering cases where // the same schema document includes, redefines, or imports // other schema documents more than once.

For purposes of this model, namespaces are objects which have identity and can thus be distinguished from each other. They have no other properties of interest. We say this with a very simple Alloy signature declaration: // Namespaces are just things that have identity. We don't // care about their properties. sig NS {}

Schema documents, on the other hand, do have an internal structure we care about. We ignore most of what can actually happen in a schema document: we don't model element or attribute declarations or type definitions, only the relations between schema documents (and schemas) defined in section 4 of the XSDL spec (, ), namely inclusion, import, and redefinition. For now we ignore the sequence of xs:include, xs:import, and xs:redefine elements, and also the possibility of multiple arcs with the same values for originating node, target node, and label. It may be worth modeling the order of calls, and multiple redundant calls, in a later variation of the model. Also, each schema document has an optional target namespace. // Schema documents have a target namespace, includes, // excludes, and imports. sig SchemaDoc { tns : lone NS, includes : set SchemaDoc, redefines : set SchemaDoc, imports : set SchemaDoc }

It's informative to check to see whether the model defined so far has any instances. To do this, we define an Alloy predicate asking for a non-empty world, and ask Alloy to run it. pred non_empty_universe(S : SchemaDoc) {} run non_empty_universe for 3

When we execute the command run non_empty_universe for 3, the Alloy Analyzer presents us with an instance of the predicate we have defined, in which no signature has more than three instances. The first instance found has just one schema document; in the graphical form generated by Alloy, it looks like this:

A one-document instance of the model

The figure shows a green oval labeled SD ($non_empty_universe_S) imports: SD includes: SD, with two arcs running out from the oval and back to the oval, labeled impors and includes. There is also a tan-colored box labeled NS.

The world has one schema document, SD, and one namespace, NS. SD has no target namespace, and SD both includes and imports itself.

Another more complex example may also be worth showing:

A three-document instance of the model

The figure shows three green oval labeled SD0, SD1, and SD2. Several arcs run from oval to oval, as described in the text. There is also a tan-colored box labeled NS.

There are three schema documents, none with a target namespace: SD0 both includes and redefines SD2. SD1 both includes and redefines both itself and SD0. SD2 includes SD1, redefines SD1, and imports SD0, SD1, and itself.

Notice that neither of these two instances actually represents a legal schema: the self-imports violate the rule that the namespace being imported must differ from the schema document's target namespace. Other instances generated by the model so far violate the rules about namespace matching for inclusions and redefinitions.

We can define a predicate which is true of a schema document S just in case S follows the basic rules for namespaces in schema composition: If S includes a document A, then either their target namespaces match, or else A has no target namespace. (Here and in the other rules, an absent target namespace matches another target namespace, and does not match an actual target namespace.) In Alloy, this can be expressed as all SI : S.includes | (SI.tns = S.tns or no SI.tns) for the case where S has a target namespace,A literal paraphrase of the Alloy expression may be helpful: For all SI in the set S.includes, it is the case that SI.tns = S.tns or else that there exists no namespace S.tns (S.tns is the empty set). and as all SI : S.includes | (no SI.tns) otherwise. A paraphrase: For all SI in the set S.includes, it is the case that there is nothing in the set SI.tns. If S redefines a document A, then either their target namespaces match, or else A has no target namespace. (Here and in the other rules, an absent target namespace matches another target namespace, and does not match an actual target namespace.) The Alloy expression is just as for inclusions, if we substitute redefines for includes. If S imports a document A,More exactly, the XSDL spec would say that S imports a namespace N, and identifies A as a schema document for that namespace; for purposes of this paper, we can say S imports A for this. then target namespaces differ. In Alloy: all SI : S.imports | (ST.tns != S.tns or no SI.tns) (if S has a target namespace) or as all SI : S.imports | (some SI.tns) (otherwise). Putting these together, we have the Alloy definition of the nsok (namespace OK) predicate: pred nsok[S : SchemaDoc] { some S.tns implies ( (all SI : S.includes + S.redefines | (SI.tns = S.tns or no SI.tns)) and (all SI : S.imports | (SI.tns != S.tns or no SI.tns)) ) no S.tns implies ( no S.(includes+redefines).tns and (all SI : S.imports | some SI.tns) ) } run nsok for 3

Armed with the nsok predicate, we can define a predicate that holds for worlds in which each schema document is namespace-OK:There might be some mileage in defining a signature for the class of namespace-OK schema documents; if we did, then all_clean might be defined this way: sig NSCleanSchemaDoc extends SchemaDoc {}{ nsok[this] } pred all_clean() { no SchemaDoc - NSCleanSchemaDoc } The body of the all_clean predicate says: the set difference of the set of schema documents and the set of namespace-OK schema documents (SchemaDoc - NSCleanSchemaDoc) is empty (there exists no member of the set). pred all_clean() { all S : SchemaDoc | nsok[S] } run all_clean for 1 run all_clean for 2 run all_clean for 3

If they are helpful in eliminating uninteresting test cases, further constraints might be defined. Some obvious candidates (not defined here now because in fact I think the cases which violate these constraints are interesting): No self-inclusion, no self-redefinition. (I.e. no cycles of length 1. Longer cycles need not and should not be forbidden, except to explore the implications of adding such a constraint to the spec. No overlap between the includes and the redefines of a given document. (Again, overlap in the transitive closure should not be forbidden, for test cases intended to explore XSDL 1.0 and its implementations, since in reality we know such cases arise and cause problems for implementations and users.)

It will be useful to generate some test cases which do not obey the namespace-OK constraints (or the others, if we later define them), but for the most part, we will be more interested in test cases which don't violate the namespace matching constraints, since they are clear and easily checked.

Recent versions of Alloy expose a Java API to the Alloy Analyzer, by means of which it's possible to perform the operations offered by the Analyzer's GUI, under program control. With the help of the Alloy team (in particular Felix Chang), it has proven straightforward to construct a Java program to load the model given above and write out XML descriptions of instances of the model. The format of these XML descriptions is described further below.

As an example, when the Alloy command run all_clean for 1 is executed, several instances of the model are generated. These are perhaps useful because they compactly exercise conditions of self-inclusion and self-redefinition. If we are satisfied that these instances provide an adequate check on processor behavior in such cases, we can then eliminate self-arcs from the model when generating later test cases. (If self-arcs may interact with code for other inclusions and redefinitions, we won't want to eliminate self-arcs from the cases with multiple schema documents.) A summary of the single-document test cases follows: m1a00.xml 0 NS, 0 SchemaDoc.   m1a01.xml 0 NS, 1 SchemaDoc. SchemaDoc$0

m1a02.xml 0 NS, 1 SchemaDoc. SchemaDoc$0 redefines: SchemaDoc$0.

m1a03.xml 0 NS, 1 SchemaDoc. SchemaDoc$0 includes: SchemaDoc$0.

m1a04.xml 0 NS, 1 SchemaDoc. SchemaDoc$0 includes: SchemaDoc$0; redefines: SchemaDoc$0.

m1a05.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 redefines: SchemaDoc$0.

m1a06.xml 1 NS, 0 SchemaDoc.

m1a07.xml 1 NS, 1 SchemaDoc. SchemaDoc$0

m1a08.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 tns: NS$0.

m1a09.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 tns: NS$0. redefines: SchemaDoc$0.

m1a10.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 tns: NS$0. includes: SchemaDoc$0.

m1a11.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 tns: NS$0. includes: SchemaDoc$0; redefines: SchemaDoc$0.

m1a12.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 includes: SchemaDoc$0; redefines: SchemaDoc$0.

m1a13.xml 1 NS, 1 SchemaDoc. SchemaDoc$0 includes: SchemaDoc$0.

It will be noted that some of these are essentially isomorphic: if the schema document has no target namespace, the presence of an unused namespace in the world makes no material difference. If the presence of such extraneous namespaces proves irritating (as it surely will, since it will double the number of cases to process without adding further information), a predicate can be added to require that all namespaces in the model be the target namespace of some schema document.

Before we attempt to generate any concrete test cases from these or other instances of this model, however, there are some other problems we need to consider; we turn to those problems in the next section.

Making interpretable test cases

The primary goal of the test cases is interoperability testing, not conformance testing. That is, we wish to be able to tell whether different processors behave the same way, or in different ways, when handling a particular test case. For this purpose it is not essential that we specify what the expected result of a given test is, only that it be possible to tell what the processor did. If the spec is clear, then either there will be a clearly expected result, or else it will be clear that the behavior is unspecified. If it's not clear what result is required by the spec for a given test, or whether a particular result is required, that test is not much good for conformance checking (except to illustrate that the spec is underspecified). But it can nevertheless be useful for interoperability testing.

XSDL 1.0 defines what PSVI should result from a given input document, given a particular set of schema components. But does not require a processor to expose to clients any particular information about what set of schema components was actually used in the validation. Nor does it require the processor to follow any particular rules for finding schema documents or other sources of schema components (aka component acquisition): at a first approximation, processors may effectively do whatever they like when it comes to schema composition. XSDL 1.1 does require that a processor document what component acquisition policies it follows (or can follow at user option), but XSDL 1.0 does not.

So one important challenge for construction of test cases is to try to ensure that from the test results it is possible to tell precisely what components were present in the validating schema. If schema document A includes schema document B, then we need to be able to tell, from the validation results, whether B was actually included or not. If schema document B is included by A and redefined by C, By phrases like B is redefined by C I mean that schema document C contains an xs:redefine element which specifies schema document B as a document to be consulted. At the moment, it doesn't matter which components of B are redefined in C; we need to choose the components and the terms of redefinition in such a way as to ensure that we can tell whether the xs:redefine was honored or not. then we need to be able to tell whether the relevant components from B have been redefined or not. And so on.

At the schema author's level, we would like to be able to answer questions like: Did the processor consult this particular schema document? Did the processor honor this particular xs:include element? Did the processor honor this particular xs:redefine element? Did the processor honor this particular xs:import element? If a schema document has been called more than once in different ways, which call was actually processed? Was the schema document included or imported? Included or redefined? Why did the processor consult this particular document? Was it because the document was named at invocation time? Pointed to from the document instance/ Pointed to from another schema document being consulted? In what order did the processor consult the various schema documents? (If a document is reachable more than once, in conflicting ways, a processor might raise an error. Or it might process the document the first time it sees a reference, and then decline to process it again when it sees the second reference, either because it believes the document has already been processed correctly or because it knows the second round of processing would lead to an error. In the latter cases, it will be useful to know whether a schema document was processed as for an include, or as for a redefinition.

It is not entirely clear whether test cases can easily be constructed which allow all of these questions to be answered. But I believe we can construct test cases which allow us to answer the following questions for any schema document, by looking at the validation results from any processor which either provides access to the entire PSVI or at least issues error or warning messages for invalid elements, as long as it does not stop work as soon as it sees the first invalid element.If we encounter processors which do stop as soon as they know that the root element is invalid, then we will need to change from the single test document outlined below to a set of 27, 54, or 81 test documents. (Let us call such processors indefatigable, since they do not tire of reporting validity problems.) Was this schema document consulted? Were the components redefined? By what other schema document were the components redefined? We do this by making each schema document in the test define a readily identifiable set of components, which will be detectable if present in the schema, but which will not cause problems by their absence. We also ensure that if any schema document redefines components from any schema document, it does so in a characteristic way which allows us to tell which redefinitions were involved. Finally, we provide a test document which provides element instances which will be valid, or invalid, depending on which components are present in the schema.

The discussion below assumes we define three (or at most three) schema documents with one, two, or three different values for their target namespaces; the principles can be extended for larger finite numbers of schema documents. The three schema documents are called A, B, and C.

Hand-created tests

Generating test cases

[Description here of dumping instances from Alloy, the XML format for instances, the XML format for test cases, and the XSLT to go from the one to the other. The test cases should be described by a catalog following the schema agreed on by the XML Schema Working Group, with the possible exception that in some cases, I do not know what the expected result should be. I'll either leave it out, or claim that the expected result is that the schema is valid and the document invalid. This section also needs a description of the test cases, and pointers to them on the server.]

Test results

[Description here of test results for specific processors for the test cases described above.]

Conclusion and further work

This paper has demonstrated that even a simple Alloy model of schema composition can be useful, both in sanity checking designs for this part of the schema spec and in generating test cases for interoperability testing (and, once the required behaviors are more clearly specifid, also for conformance testing).

The first item of further work is to complete this paper by automating the generation of test cases from Alloy instances the tabulation of results (for t0, manual inspection of the output has sufficed, but this has proven time-consuming even for t0; for larger test sets it will be impracticable) and then by generating test cases, using them to test as many XSDL 1.0 processors as is feasible, and studying the results.

Further models should also be developed (to be reported in separate papers, not this one), to capture more of the details of schema composition and allow the formulation of assertions to test possibly interesting propositions (e.g. for two schema documents in the same namespace, the order in which they are imported or included or redefined by other schema documents is immaterial; both orders produce the same result — as formulated, I expect this proposition to be false, but it expresses a central idea which I believe is true; getting the formulation accurate is a challenge with which light-weight formal methods like Alloy can assist.

Thanks are due to Daniel M. Jackson and the Alloy team for the tool itself and for assistance with questions and problems; to Liam Quin, for discussion of the issues raised in this paper; to David Ezell and Michael Kay for helpful discussions and for assistance in gathering test results.