Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UI tests extremely flaky on macOS in GitHub Actions #3057

Closed
LorenzoBettini opened this issue May 21, 2024 · 5 comments · Fixed by #3063
Closed

UI tests extremely flaky on macOS in GitHub Actions #3057

LorenzoBettini opened this issue May 21, 2024 · 5 comments · Fixed by #3063
Milestone
Release_2.36

Comments

@LorenzoBettini
Copy link
Contributor

Some examples:

Failures: 
org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 4: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_4:295->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.

Flakes: 
org.eclipse.xtend.ide.tests.compiler.ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProject
  Run 1: ActiveAnnotationsInSameProjectTest.testActiveAnnotationInSameProject:144->assertHasErrors:529->Assert.fail:89 Expected an error marker containing 'same project' on /test.project/src/mypack/Client.xtend but found Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.CircularDepsBetweenJavaAndXtendTest.testJavaSignatureDependsOnXtend
  Run 1: CircularDepsBetweenJavaAndXtendTest.testJavaSignatureDependsOnXtend:242->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 expected:<0> but was:<3>
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef:261->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_3
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_3:283->testResolvingXFunctionTypeRef:447->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXFunctionTypeRef_5:307->testResolvingXFunctionTypeRef:421->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 org.eclipse.emf.ecore.EObject cannot be resolved to a type.
  Run 2: PASS

org.eclipse.xtend.ide.tests.compiler.ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference
  Run 1: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:246->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 2: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:246->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 myannotation.MyAnnotation cannot be resolved to a type.
  Run 3: ResolvingCrossReferenceDuringIndexingTest.testResolvingXtendAnnotationReference:225->assertNoErrorsInWorkspace:460->Assert.assertFalse:65->Assert.assertTrue:42->Assert.fail:89 The method or field Assert is undefined
  Run 4: PASS

org.eclipse.xtend.ide.tests.macros.JdtBasedProcessorProviderTest.testClassLoaderSeesAllUpstreamProjects
  Run 1: JdtBasedProcessorProviderTest.testClassLoaderSeesAllUpstreamProjects:137 Workspace contained errors: 
 - Lib.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
 - A.xtend:1 - Couldn't find the mandatory library 'org.eclipse.xtext.xbase.lib' 2.8.0 or higher on the project's classpath.(org.eclipse.xtend.ide.xtend.check.fast)
 - A.xtend:3 - annotation.MyAA cannot be resolved to an annotation type.(org.eclipse.xtend.ide.xtend.check.fast)
  Run 2: PASS


Tests run: 3121, Failures: 1, Errors: 0, Skipped: 52, Flakes: 7

Maybe there's some missed synchronization with project refresh and building.
Even when the build succeeds, there are lots of flakes.

I can try increase the number of retries.
@LorenzoBettini
Copy link
Contributor Author

Xtend IDE tests are the favorite, but sometimes it happens much before, in Xtext UI tests:

Results:

Failures: 
org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_01
  Run 1: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testBug463215_01:114->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testBug463215_02
  Run 1: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testBug463215_02:125->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_01
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_01:136->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_02
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_02:147->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_03
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_03:155->Assert.assertEquals:146->Assert.assertEquals:117 expected:<[org.eclipse.xtext.Xtext
org.eclipse.xtext.common.Terminals]> but was:<[]>

org.eclipse.xtext.xtext.ui.editor.contentassist.XtextContentAssistTest.testCompletionOnDatatypeReference_04
  Run 1: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 2: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 3: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]
  Run 4: XtextContentAssistTest.testCompletionOnDatatypeReference_04:161->Assert.fail:89 No such proposal: Terminals - org.eclipse.xtext.common.Terminals Found: [@, @Deprecated, @Exported, @Final, @Override]


Tests run: 267, Failures: 6, Errors: 0, Skipped: 1

@cdietrich
Copy link
Member

See also eclipse/mwe#270

@LorenzoBettini
Copy link
Contributor Author

I guess yes...

for Xtend UI tests I can increase the test retries.

For XtextContentAssistTest maybe that won't help, because the project is created only once and I guess once the build has been done it won't be retried:

	@Before
	public void doSetupProject() throws Exception{
		if(project == null){
			with(doGetSetup());
			PluginProjectFactory projectFactory = getInjector().getInstance(PluginProjectFactory.class);
			projectFactory.setProjectName(TEST_PROJECT);
			projectFactory.addFolders(Collections.singletonList("src"));
			projectFactory.addBuilderIds(XtextProjectHelper.BUILDER_ID, JavaCore.BUILDER_ID, "org.eclipse.pde.ManifestBuilder",
					"org.eclipse.pde.SchemaBuilder");
			projectFactory.addProjectNatures(
					XtextProjectHelper.NATURE_ID, JavaCore.NATURE_ID, "org.eclipse.pde.PluginNature");
			projectFactory.addRequiredBundles(Lists.newArrayList("org.eclipse.xtext"));
			project = projectFactory.createProject(new NullProgressMonitor(), null);
			get(SyncUtil.class).waitForBuild(new NullProgressMonitor());
		}
	}

maybe moving the waitForBuild outside the if?

On a side note, in my projects things improved a lot concerning flakiness in these situations, if not completely fixed, by stopping creating projects programmatically and by importing existing ad-hoc projects for testing, e.g., https://github.com/LorenzoBettini/edelta/blob/master/edelta.parent/edelta.ui.tests/src/edelta/ui/tests/EdeltaContentAssistTest.java#L57, using https://github.com/LorenzoBettini/edelta/blob/master/edelta.parent/edelta.ui.tests/src/edelta/ui/tests/utils/ProjectImportUtil.java

Moreover, I wonder whether project creation in projectFactory happen in a workspace operation in a sort of synchronized way.

This was referenced May 22, 2024
Closed
Merged
@LorenzoBettini LorenzoBettini added this to the Release_2.35 milestone May 23, 2024
@LorenzoBettini
Copy link
Contributor Author

I'm re-opening it: there are still failures on macOS :'(

let's see whether increasing the retries makes things better, and then we'll investigate further...

This was referenced May 23, 2024
Merged
Open
@cdietrich cdietrich modified the milestones: Release_2.35, Release_2.36 May 27, 2024
@LorenzoBettini
Copy link
Contributor Author

I'm closing this, since I don't see that anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Release_2.36
Development

Successfully merging a pull request may close this issue.

DetectVMInstallationsJob.disabled as system property of tycho-surefire LorenzoBettini/xtext
2 participants
@cdietrich @LorenzoBettini