Integrate Xtext editor with EAXML serialization

[Wilson008]

The EATOP plugin for the EAXML serialization has to be integrated, so that either

• on saving the *.eatxt file at the same time an *.eaxml file is preserved;
• or a dedicated action ("save as EAXML") to preserve the *.eaxml file on demand

A flow-chart of serialization:

[joerg-holtmann]

My suggestion here is (without much background knowledge, so no warranty that this is the right way):

• Integrate the (de-)serialization plugin from EATOP
• Include an action (i.e., an UI contribution), which explicitly calls a "file save as EAXML" wizard which lets the user choose where to save the .eaxml file. After completing the wizard, the action then uses the (potentially adapted) serialization plugin from EATOP to serialize the EMF ResourceSet (which is parsed from the .eatxt file) as EAXML into the file specified by the user.
• The same procedure could then also be used to read an EAXML file for roundtripping, but maybe some rendering features have to be added in case Xtext cannot do this on its own.

[Wilson008]

Something related in this link: https://jevopisdeveloperblog.blogspot.com/2011/03/implement-tostring-with-xtexts.html

[steghoja]

I am not a fan of the toString() method since it means we need to change what we have generated. Since we want to serialise/deserialise whole models, we should use the existing capabilities of EMF for this. I have already implemented this in org.bumble.eastadl.simplified.ui.handlers.ImportFromEaxmlHandler for the conversion from EAXML to eatxt, but as I said, I have problems when EMF tries to resolve the references.

[steghoja]

There has been some progress and some riddles. Here's a summary:

Originally, deserialisation to EAXML failed since EMF did not recognise the URIs used by EAST-ADL with the protocol ea and through MalformedURIExceptions. Apparently, the EastADLURIFactory was not used to dereference the URIs. This was solved by using Sphinx' ScopingResourceSetImpl as the resource set. That solves the issue for unknown reasons.

However, now serialisation to EAtxt fails with the following error:

Caused by: java.lang.RuntimeException: No EObjectDescription could be found in Scope RangeableValueType.baseRangeable for EANumerical
Semantic Object: EAXML.topLevelPackage[0]->EAPackage''.element[0]->RangeableValueType''
URI: file:/Users/janste/git/east-adl-textual-syntax/examples/org.bumble.eastadl.simplified.examples/BBW_4Wheel_Bumble.eatxt
EStructuralFeature: ::RangeableValueType.baseRangeable
	at org.eclipse.xtext.serializer.diagnostic.ISerializationDiagnostic$ExceptionThrowingAcceptor.accept(ISerializationDiagnostic.java:132)
	at org.eclipse.xtext.serializer.tokens.CrossReferenceSerializer.getCrossReferenceNameFromScope(CrossReferenceSerializer.java:139)
	at org.eclipse.xtext.serializer.tokens.CrossReferenceSerializer.serializeCrossRef(CrossReferenceSerializer.java:112)
	at org.eclipse.xtext.serializer.acceptor.SequenceFeeder.getToken(SequenceFeeder.java:483)
	at org.eclipse.xtext.serializer.acceptor.SequenceFeeder.accept(SequenceFeeder.java:246)

After updating the EastAdlSimplifiedRuntimeModule as described in https://www.eclipse.org/forums/index.php/t/357707/ the scope now looks like this:

SimpleScope[DataTypesPackage.MyFloatDatatype, DataTypesPackage.MyTimeDatatype] -> NULLSCOPE

This contains DataTypesPackage.MyFloatDatatype which is actually what we’re looking for.

When Xtext then looks for the right EObjectDescription, it looks for a target which is this:

org.eclipse.eatop.eastadl22.impl.EANumericalImpl@4ceeba96 (eProxyURI: ea:/%23platform:/#/DataTypesPackage/MyFloatDatatype?type=EANumerical)

The elements in the scope look like this:

[org.eclipse.eatop.eastadl22.impl.EANumericalImpl@59bcab83 (shortName: MyFloatDatatype) (category: , uuid: 7767d152-b6f0-411d-1) (name: ) (text: , uri: <unset>) (max: , min: ), org.eclipse.eatop.eastadl22.impl.EANumericalImpl@27c243a3 (shortName: MyTimeDatatype) (category: , uuid: 7767d152-b6f0-411d-2) (name: ) (text: , uri: <unset>) (max: , min: )]

The first one is probably the one we want. The comparison is based on the URI. But instead of using the URI ea:/%23platform:/#/DataTypesPackage/MyFloatDatatype?type=EANumerical, it uses the URI file:/Users/sadsad/git/east-adl-textual-syntax/examples/org.bumble.eastadl.simplified.examples/BBW_4Wheel_Bumble.eatxt#//@topLevelPackage.0/@element.5 which is not what we want and why Xtext does not find the right class!

So we need to get Xtext to use the URI provided by EastADLURIFactory in the scope. Somehow...

[steghoja]

Creating an EAXML file from an eatxt file works right now. However, the EAXML file contains tags such as this:

<CATEGORY xsi:nil="true"/>

The EAXML files we received as examples would instead use this format:

<CATEGORY><CATEGORY/>

We need to check if this is a problem or if we can safely ignore this. So far, EATOP seems to accept files that contain tags of this form without issue.

[steghoja]

I have come a little further now. Using the same approach as EAST-ADL uses, I have now created a new EatxtResource which inherits from XtextResource and overwrites EObject getEObject(String) and String getURIFragment(EObject). This means that at least the fragment part of the URIs (e.g., //@topLevelPackage.0/@element.5) is now resolved as expected by EAST-ADL (i.e., as /DataTypesPackage/MyFloatDatatype?type=EANumerical,). However, the comparison still fails since Xtext uses file:/ URLs and the EMF model contains ea:/ ones...

[steghoja]

More progress. In 8d3cca3, I added custom scope to accommodate different URI schemes. When serialising an EMF model that was loaded from an EAXML file to an EAtxt file, Xtext uses file as the scheme for the EObject URIs. The reason is that Xtext is not aware of EastADLURIFactory and there is no simple way (that I could find) to make it aware of that. Instead, we are now using a specialised implementation of Scope, that compares the URI fragments if the URI of the EObject that we look for in the scope has ea as its scheme. This allows us to find the correct element since East-ADL uses a global namespace (confirm!).

This is not the end of the story, though. Several issues are left:

• For some reasons, some of the URIs that are being compared have a platform:/ in there which shouldn't really be the case.
• I also get an IllegalStateException when trying to serialise more complex examples:

 java.lang.IllegalStateException
	at org.eclipse.xtext.serializer.sequencer.AbstractSyntacticSequencer.enterAssignedParserRuleCall(AbstractSyntacticSequencer.java:354)
	at org.eclipse.xtext.serializer.acceptor.SequenceFeeder.acceptEObjectRuleCall(SequenceFeeder.java:325)
	at org.eclipse.xtext.serializer.acceptor.SequenceFeeder.acceptRuleCall(SequenceFeeder.java:354)
	at org.eclipse.xtext.serializer.acceptor.SequenceFeeder.accept(SequenceFeeder.java:265)

• Finally, the serialised EAtxt does not contain line breaks and does not seem to honour the whitespace.

[steghoja]

Even after introducing line breaks and whitespace manually (see below), the eatxt file is not valid. There are two problems:

1. The parser does not like the empty attributes (category, name, etc.).
2. The UUIDs are not parsed. That's because they map to the Xtext standard terminal ID which does not contain hyphens and considers IDs that start with a number incorrect. We would rather need a rule such as:

terminal UUID:
	'^'? ('a'..'f'|'A'..'F'|'$'|'_'|'0'..'9') ('a'..'f'|'A'..'F'|'$'|'_'|'0'..'9'|'-')*;

Here's the sanitized generated eatxt file:

EAPackage DataTypesPackage
  category
  uuid 03
  name
  RangeableValueType AA_PercentType
    category
    uuid ac61934c-8851-4742-acde-2f41e401febb
    name
    text
    accuracy 1.0
    resolution 1.0
    significantDigits 3
    baseRangeable "DataTypesPackage.MyFloatDatatype"
  EANumerical MyFloatDatatype
    category
    uuid 7767d152-b6f0-411d-1
    name
    text 
    max
    min
    unit "DataTypesPackage.MyTimeUnit" 
  Unit MyTimeUnit
    category uuid 7767d152-b6f0-411d-3
    name
    factor 1.0
    symbol ms
    offset 0.0
    quantity "DataTypesPackage.MyTimeQuantity"
  Quantity MyTimeQuantity
    category
    uuid 7767d152-b6f0-411d-4
    name
    amountOfSubstanceExp 0
    electricCurrentExp 0
    lengthExp 0
    luminousIntensityExp 0
    massExp 0
    thermodynamicTemperatureExp 0
    timeExp 1

[steghoja]

It's indeed the formatter that needs to be extended. I have added the code below (took me lots of debugging to figure out) and now have at least some linebreaks in the the generated code. What I do not know yet is how to deal with the attributes. The code I am currently using fails for most of them. Maybe because they are empty? In any case, the regionFor.feature() method returns null, even though the feature exists.

Itemis has an example for their whitespace-aware language but it's not documented and I find it hard to map to our much more complicated (and nested) language.

	def dispatch void format(EAElement obj, extension IFormattableDocument doc) {
		if (obj === null) {
			return;
		} else {
			obj.append[newLine]
			obj.regionFor.feature(Eastadl22Package.eINSTANCE.getEAElement_Name()).surround[indent].append[newLine]  
			if (obj instanceof Identifiable) {
				obj.regionFor.feature(Eastadl22Package.eINSTANCE.getIdentifiable_Category()).surround[indent].append[newLine]
				obj.regionFor.feature(Eastadl22Package.eINSTANCE.getIdentifiable_Uuid()).surround[indent].append[newLine]
			}
		}
		obj.eContents.forEach[format]
	}
	
	def dispatch void format(EAPackageableElement obj, extension IFormattableDocument doc) {
		if (obj === null) {
			return;
		} else {
			obj.surround[indent].append[newLine]
		}
		obj.eContents.forEach[format]
	}

[steghoja]

The current version is able to convert correct EAXML to EAtxt and vice versa. There are some remaining issues:

• References should not be validated (Disable reference validation when serialising as EAtxt #31 )
• Attributes should all be optional (Make all attributes and references optional #30)
• We need to check if instanceref works as intended (may be related to Support adjustable instanceRef #3?)